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PREFACE 


As  a  private  non-profit  corporation.  New  American  Schools  (NAS)  began  in  1991 
to  fund  the  development  of  designs  aimed  at  transforming  entire  schools  at  the 
elementary  and  secondary  levels.  Having  completed  competition  and  development 
phases,  NAS  currently  is  scaling-up  its  designs  across  the  nation.  During  the  phase 
when  NAS  was  committed  to  implementing  its  designs  across  schools  within  partnering 
jurisdictions,  RAND's  research  assessed  the  impact  of  NAS  designs  on  classroom 
practice  and  student  achievement  in  a  sample  of  schools  in  a  high-poveriy  district 
during  the  1997-1998  and  1998-1999  schools  years. 

This  is  a  report  aimed  at  those  who  want  to  better  imderstand  the  expanding  area 
of  whole-school,  or  comprehensive,  school  reform  and  its  effects  on  teaching  and 
learning  within  high-stakes  accoimtability  environments. 

The  RAND  evaluation  of  NAS  schools  has  spanned  several  years.  To  date,  RAND 
studies  about  New  American  Schools  include: 

Implementation  in  a  Longitudinal  Sample  of  New  American  Schools:  Four  Years  into 
Scale-Up,  by  Sheila  Nataraj  Kirby,  Mark  Berends,  and  Scott  Naftel.  (DRU- 
2468-EDU). 

The  Relationship  Between  Implementation  and  Achievement:  Case  Studies  of New 

American  Schools,  by  JoAn  Chun,  Brian  GUI,  and  Jodi  HeUbrunn.  (DRU- 
2562-EDU). 

"Reforming  Whole  Schools:  Challenges  and  Complexities,"  by  Mark  Berends, 
Susan  Bodilly,  and  SheUa  Nataraj  Kirby.  Forthcoming  in  Bringing  Equity 
Back,  edited  by  J.  Petrovich  and  A.  W.  Wells. 

"Implementation  in  New  American  Schools:  A  Longitudinal  Analysis"  by  SheUa 
Nataraj  Kirby,  Mark  Berends,  and  Scott  Naftel.  To  appear  in  XXX 
(Temple  Lab  book). 

Implementation  and  Performance  in  New  American  Schools:  Three  Years  into  Scale-up 
by  Mark  Berends,  Sheila  N.  Kirby,  Scott  Naftel,  and  Christopher 
McKelvey,  2001  (MR-1145-EDIJ). 

New  American  Schools '  Concept  of  Break  the  Mold  Designs:  How  Designs  Evolved 
Over  Time  and  Why  by  Susan  BodiUy,  2001  (DRU-2143-EDU). 
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"Teacher-Reported  Effects  of  New  American  Schools'  Designs:  Exploring 
Relationships  to  Teacher  Background  and  School  Context"  by  Mark 
Berends  in  Educational  Evaluation  and  Policy  Analysis,  2000, 22(1),  65-82. 
"Necessary  District  Support  for  Comprehensive  School  Reform"  by  Susan  J. 
Bodilly  and  Mark  Berends.  Pp.  111-119  in  Hard  Work  for  Good  Schools: 
Facts  Not  Fads  in  Title  1  Reform,  edited  by  Gary  Orfield  and  Elizabeth  H. 
DeBray.  Boston,  MA:  Civil  Rights  Project,  Harvard  University,  1999. 
Assessing  tlw  Progress  of  New  American  Schools:  A  Status  Report  by  Mark  Berends, 
1999  (MR-1085-ED). 

Lessons  from  New  American  Schools'  Scale-up  Phase:  Prospects  for  Bringing  Designs 
to  Multiple  Schools  by  Susan  J.  Bodilly,  1998  (MR-1777-NAS). 

New  American  Schools  after  Six  Years  by  Thomas  K.  Glennan,  Jr.,  1998  (MR-945- 
NASDQ. 

Funding  Comprehensive  School  Reform  by  Brent  R.  Keltner,  1998,  (IP-175-EDU). 
Reforming  America's  Schools:  Observations  on  Implementing  “Whole  School  Designs" 
by  Susan  J.  Bodilly  and  Thomas  K.  Glennan,  1998,  (RB-8016-EDU). 

Lessons  from  New  American  Schools  Development  Corporation 's  Demonstration  Phase 
by  Susan  J.  Bodilly,  19%  (MR-729-NASDC). 

Reforming  and  Conforming:  NASDC  Principals  Discuss  School  Accountability 
Systemshy  Karen  Mitchell,  19%  (MR-716-NASDC). 

"Lessons  Learned  from  RAND' s  Formative  Assessment  of  NASDC's  Phase  2 
Demonstration  Effort"  by  Susan  J.  Bodilly.  Pp.  289-324  in  Bold  Plans for 
School  Restructuring:  The  New  American  Schools  Designs,  edited  by  Sam 
Stringfield,  Steven  Ross,  and  Lana  Smith.  Mahwah,  NJ:  Lawrence 
Erlbaiun  Associates,  19%. 

Designing  New  American  Schools:  Baseline  Observations  on  Nine  Design  Teams  by 

Susan  J.  Bodilly,  Susanna  Purnell,  Kimberly  Ramsey,  and  Christina 
Smith,  1995  (MR-598-NASDQ. 

Fvmding  for  this  research  was  provided  under  a  contract  with  NAS  and  supported 
by  the  Ford  Foundation  and  another  donor.  This  report  was  written  under  the  aegis  of 
RAND  Education. 
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SUMMARY 


A  decade  ago.  New  American  Schools  (NAS)  laxmched  an  ambitious  effort  for 
whole-school  reform  to  address  the  perceived  lagging  achievement  of  American 
students  and  the  lackluster  school  reform  attempts  that  produced  so  little  meaningful 
change.  As  a  private  non-profit  organization,  NAS  set  out  to  help  schools  and  districts 
significantly  raise  the  achievement  of  large  numbers  of  students  by  offering  whole- 
school  designs  and  design-based  assistance  during  the  implementation  process.  NAS  is 
currently  in  the  scale-up  phase  of  its  effort.  Its  designs  are  being  widely  diffused  in 
partnering  jurisdictions  across  the  nation. 

The  purpose  of  this  report  is  to  examine  the  conditions  of  NAS  classrooms 
compared  with  non-NAS  classrooms  and  to  study  the  relationships  between  classroom 
conditions  and  student  achievement.  The  focus  is  on  the  conditions  in  the  district, 
schools,  and  classrooms  that  promote  or  inhibit  design  implementation  and  changes  in 
teaching  and  learning. 

Specifically,  this  study  addresses  the  following  questions: 

•  Do  the  NAS  designs  extend  beyond  changes  in  school  organization  and 
governance  and  permeate  classrooms?  Do  NAS  teachers  and  students 
interact  with  each  other  and  subject  materials  in  ways  that  reflect  the 
innovative  curricular  and  instructional  approaches  of  the  design  teams? 

•  What  factors  at  the  district,  school,  and  classroom  level  are  related  to 
implementation  of  designs  and  changes  in  classroom  instruction? 

•  Do  the  NAS  designs  and  the  design-based  assistance  provided  lead  to 
changes  in  curriculum  and  instruction  and  thereby  achievement? 

Our  results  provide  useful  information  to  NAS,  the  design  teams,  policymakers, 
and  the  public  to  improve  efforts  aimed  at  whole-school  restructuring  and 
comprehensive  school  reform,  particularly  in  high-poverty  settings. 

THE  ANALYSIS  SAMPLE 

The  schools  included  in  this  study  were  those  involved  in  the  early  stages  of  the 
district' s  partnership  with  NAS.  The  NAS  designs  being  implemented  in  this  district  at 
the  time  of  this  study  included  Co-NECT,  Expeditionary  Learning  Outward  Bound 
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[ELOB],  Modem  Red  Schoolhouse  [MRSH],  and  Success  for  All/Roots  &  Wings 
[SFA/RW]i 

We  gathered  a  variety  of  data,  including;  principal  and  teacher  surveys 
conducted  at  the  end  of  the  1997-1998  and  1998-1999  school  years;  interviews  with 
district  staff,  design  team  leaders,  local  facilitators,  principals,  and  teachers;  classroom 
observations;  illustrative  examples  of  student  work;  data  provided  by  the  district  on  test 
scores  and  student  and  teacher  demographic  characteristics;  and  achievement  data  from 
a  supplementary  test  administered  to  students  (Stanford-9  reading).  In  addition,  this 
study  relies  on  other  RAND  research  on  NAS  that  included  site  visits  to  schools  and 
school  districts  to  gather  information  about  district  and  school  administrators'  and 
teachers'  reports  of  the  progress  of  the  NAS  initiative  (Berends  and  Kirby  et  al.,  2001; 
BodiUy,  1998, 2001). 

RAND  collected  these  data  on  a  sample  of  4‘*’  grade  teachers  and  their  students 
during  two  school  years.  For  analyzing  changes  in  teacher  practice  between  the  1997- 
1998  and  1998-1999  school  years,  we  relied  on  a  longitudinal  sample  of  40  teachers.  In 
1997-1998,  we  were  also  able  to  observe  and  gather  classroom  artifacts  from  12  teachers 
in  NAS  and  non-NAS  schools,  and  in  the  following  year,  we  were  able  to  gather  such 
data  from  about  19  teachers.  The  analysis  sample  relating  classroom  conditions  to 
student  achievement  consisted  of  over  60  teachers  and  roughly  850  students,  but  we  also 
compare  our  results  with  aU  elementary  schools  and  fourth  grade  teachers  and  students 
within  the  district. 

IMPORTANT  LIMITATIONS 

There  are  several  important  limitations  of  this  research  that  need  to  be  kept  in 
mind.  First,  for  most  design  teams,  the  sample  of  schools  analyzed  in  this  study  were 
the  first  buildings  to  which  they  provided  implementation  assistance  on  a  fee-for- 
service  basis.  Many  changes  have  been  made  to  both  the  designs  and  the  assistance 
provided  as  the  teams  and  the  schools  have  gained  experience  (see  BodiUy,  2001).  Thus, 

1  While  SFA  has  been  aroxmd  for  the  past  couple  of  decades,  NAS  provided  funding  to  die 
Success  for  All  Foimdation  to  develop  and  implement  Roots  &  Wings,  which  not  oidy  includes 
die  reading  program  of  SFA,  but  also  builds  in  odier  curricular  programs  such  as  Math  Wings  and 
WorldLab.  San  Antonio  schools  were  only  implementing  the  SFA  component  of  SFA/RW  during 
the  time  of  this  study.  Because  the  Success  for  AU  Foundation  considers  aU  SFA  schools  as 
potential  RW  schools  and  because  NAS  provided  funding  for  RW,  we  refer  to  this  design  as 
SFA/RW. 
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when  interpreting  the  findings  in  this  report,  it  is  important  to  note  the  unique  features 
of  the  population  of  schools  we  studied. 

Second,  the  fact  that  designs  are  evolving  over  time  as  design  teams  gain 
experience  and  adapt  to  local  contexts  may  make  future  implementations  more 
successful.  However,  this  is  still  an  open  question,  and  some  of  our  research  calls  this 
into  question  (Kirby,  Berends,  and  Naftel,  2001).  Certainly,  additional  empirical  work 
would  be  helpful  to  address  this  issue. 

Third,  the  district  context  in  which  these  schools  were  implementing  has  changed 
dramatically  since  the  time  the  data  for  this  report  were  collected.  Similar  to  many 
urban  contexts,  the  superintendent  and  key  staff  have  moved  on  to  other  positions,  and 
some  new  district  policies  have  been  put  in  place.  Whether  these  are  more  or  less 
conducive  to  NAS  design  implementation  is  beyond  tire  scope  of  the  current  analysis. 

NAS  WITHIN  A  REFORM-MINDED  URBAN  SCHOOL  DISTRICT 

As  is  evident  in  our  description  of  these  schools  and  classrooms  within  an  urban 
school  district,  NAS  partnered  with  schools  and  jurisdictions  that  are  predominantly 
low-performing,  urban,  high  poverty,  and  high  minority  (Berends,  Kirby,  Naftel,  and 
McKelvey,  2001;  Berends,  1999). 

While  New  American  Schools  (NAS)  was  busy  starting  up  in  July  of  1991,  the  San 
Antonio  school  district  struggled  to  raise  its  students'  achievement  levels  and  meet  the 
challenges  it  faced.  At  the  time,  productive  communication  proved  problematic,  as  did 
the  effective  utilization  of  district  staff.  Much  energy  was  expended  on  the  management 
of  day-to-day  organizational  affairs.  According  to  several  central  office  administrators, 
instructional  practice  was  too  often  last  addressed. 

When  the  new  superintendent  came  on  board,  there  were  significant  changes  in 
tire  district  that  occurred.  The  superintendent  proceeded  to  draw  up  five  district  goals, 
increasing  student  achievement,  fostering  collaboration  and  communication, 
strengthening  parent  and  community  involvement,  building  an  infrastructure  for 
professional  development,  and  providing  appropriate  school  facilities  to  all  students. 

While  restructuring  instructional  leadership,  rethinking  the  dehvery  and  content 
of  professional  development,  introducing  instructional  strategies  to  teachers,  pushing 
state  standards,  and  refocusing  the  district' s  attention  on  instruction  and  student 
achievement,  San  Antonio  district  administrators  simultaneously  reviewed  national 
reform  efforts  and  programs.  Central  office  administrators  seriously  examined  and 
eventually  decided  to  implement  the  reform  ideas  of  NAS— particularly  NAS'  approach 
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to  comprehensive  school  reform.  Convinced  that  the  designs  could  play  an  important 
role  in  the  district's  efforts  to  bring  about  increased  student  achievement,  the  NAS 
designs  became  an  important  piece  of  the  reform  package  in  the  district.  Viewing  NAS 
designs  as  the  framework  and  glue  to  tie  the  multiple  district  initiatives  together,  the 
central  office  expected  to  monitor  the  progress  of  design  implementation  and  support 
the  schools  in  their  efforts. 

NAS  IMPLEMENTATION  IN  SCHOOLS  FACING  HIGH-STAKES 
ACCOUNTATILITY 

The  district  introduced  the  NAS  initiative  to  schools  with  hopes  that  the 
marketed  "break-tiie-mold"  designs  would  provoke  teachers  and  administrators  to 
engage  enthusiastically  in  comprehensive  school  reform.  The  thought  was  that  an 
external  model  provider  would  be  more  successful  at  pushing  and  sustaining  change 
than  die  central  office  could  ever  be  alone.  The  district  had  every  intention  of  fully 
supporting  its  NAS  schools  in  all  ways— including  professional  development,  site  based 
facilitators  (called  Instructional  Guides),  and  other  resources  for  the  schools  to 
implement  the  designs. 

Given  this,  in  1996,  when  NAS  designs  were  first  introduced,  one  might  have 
expected  that  in  time,  design  schools  would  look,  feel,  and  in  some  ways  function 
differently  from  one  another  as  well  as  from  schools  that  had  elected  not  to  take  on 
comprehensive  school  reform  models. 

However,  a  closer  examination  of  both  whole-school  and  individual  classroom 
activities  of  both  whole-school  and  individual  classroom  activities  tended  to  reveal  a 
more  complicated  story.  In  our  research  we  focused  on  the  challenging  educational 
environments  that  these  schools  faced,  the  high  stakes  accoimtability  system  in  which 
they  operated,  the  process  for  adopting  NAS  designs,  support  for  implementation 
including  training  and  professional  development,  principal  leadership  and  school 
resources  devoted  specifically  to  NAS  design  implementation,  and  teacher  collaboration 
and  support  of  the  NAS  designs. 

The  press  to  improve  test  scores  on  the  state  tests  (Texas  Assessment  of  Academic 
Skills  p’AAS])  was  clearly  evident  during  the  time  of  our  study.  For  instance,  in 
addition  to  the  NAS  designs,  the  district  established  an  Office  of  Curriculum  and 
Instruction  responsible  for  developing  a  sequential,  standards  aligned  curriculiun  across 
grade  levels  in  all  schools  throughout  the  district.  The  subjects  covered  on  the  Texas 
Assessment  of  Academic  Skills  (TAAS),  namely  mathematics  and  reading,  were  given 
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primary  attention.  Thus,  schools  were  not  only  exposed  to,  but  required  to,  implement 
many  ideas  at  once,  naturally  resulting  in  some  confusion  and  resistance  on  the  part  of 
school  staff. 

To  address  the  demands  of  the  TAAS,  the  district  implemented  specific 
mathematics,  reading,  and  language  arts  programs  in  addition  to  the  NAS  designs.  In 
die  spring  of  1996,  aU  schools  were  implementing  Everyday  Mathematics—  developed  by 
the  University  of  Chicago  School  Mathematics  Project.  The  district  expectations  were 
that  all  schools  throughout  the  district  would  follow  a  similar  pace,  and  the  district 
developed  pacing  guides  to  ensure  that  this  would  happen.  In  addition,  San  Antonio 
elementary  schools  implementing  a  reading  program  that  involved  a  90-minute  reading 
block.  By  the  1998-1999  school  year,  elementary  schools,  district  wide,  were  not  only 
expected  to  schedule  two  90-minute  blocks  of  uninterrupted  instructional  time  for 
reading  and  math,  respectively,  teachers  were  required  to  manage  time  within  these 
blocks  in  prescribed  ways.  Though  not  to  the  same  degree,  the  district  structured 
language  arts  activities  (spelling,  grammar,  and  writing)  as  well,  totaling  approximately 
70  minutes  of  instruction  time  per  day.  Thus,  roughly  three  hours  of  instructional 
activities  were  mapped  out  for  all  the  district' s  elementary  school  teachers  to  follow 
(SFA/RW  teachers  were  exempt  from  the  district  reading  program). 

In  addition,  lack  of  time  during  the  school  day —a  chronic  issue— became  even 
more  problematic  in  light  of  teachers'  needs  to  balance  TAAS  preparation  with  other 
instruction.  Many  teachers  reportedly  coped  with  the  multiple  demands  on  their  time 
by  putting  aside  otiier  activities  to  focus  almost  exclusively  on  TAAS  as  the  test  dates 
grew  closer. 

Within  this  context,  the  district  provided  a  substantial  amount  of  professional 
development  to  teachers.  Much  of  the  in-service  training  revolved  around  the  district' s 
reading  and  mathematics  initiatives.  Because  NAS  teachers  were  obligated  to  attend  as 
many  of  these  various  in-services  as  their  colleagues  in  non-NAS  schools,  the  amount  of 
training  activities  served  only  to  heighten  frustrations.  All  of  the  designs  except 
SFA/RW  required  teachers  to  develop  units  and  write  curriculum.  While  encouraging 
schools  to  implement  NAS  designs,  the  district  simultaneously  constrained  their  ability 
to  do  so  by  tellmg  teachers  what  to  teach  and  how. 

The  district  and  design  teams  did  not  tend  to  coordinate  their  efforts  with  respect 
to  professional  development,  so  teachers  were  left  on  their  own  to  merge  the 
information  they  received  from  each.  This  was  not  easily  done  without  modifying  the 
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essence  of  each  design.  Not  only  did  ihis  effort  burden  teachers'  workload,  it  also  led  to 
confusion  as  to  what  to  prioritize. 

Design  teams  provided  another  important  source  of  support  for  implementation. 
Each  design  team  aims  to  provide  schools  and  teachers  widr  resources  to  assist  in 
implementation,  especially  in  terms  of  communication  between  design  team  members 
and  school  staff,  and  design-related  professional  development.  For  instance,  by  1999  a 
relatively  high  proportion  of  teachers  in  the  NAS  schools  (88  percent)  agreed  that  then- 
respective  design  teams  had  clearly  communicated  "its  program  to  school  staff  so  that  it 
could  be  well-implemented." 

Part  of  this  communication  involves  teacher  training  by  design  teams.  However, 
according  to  teachers  across  design  schools,  there  was  little  regular,  consistent  assistance 
provided.  Over  time,  there  was  even  less  contact  between  teachers  and  their  respective 
design  representatives.  In  large  part,  this  had  to  do  with  the  fact  that  these 
representatives  serviced  numerous  schools,  making  it  difficult  for  them  to  be  attentive 
to  any  one.  It  also  appears  that  from  the  start,  strong  relationships  rarely  were 
established,  making  it  unlikely  that  teachers  would  rely  on  their  respective  design 
representatives  for  external  technical  support  and  assistance.  In  some  schools,  design 
representatives  turned  over,  disrupting  what  rapport  had  been  established. 

CLASSROOM  CONDITIONS  IN  NAS  AND  NON-NAS  SCHOOLS 

Our  analyses  revealed  few  differences  in  teacher  perceptions  of  instructional 
environments  between  NAS  and  non-NAS  schools.  Some  changes  were  evident.  For 
example,  teachers  in  NAS  schools  reported  mstructional  strategies  and  classroom 
practices  that  could  be  categorized  as  reform-like  (e.g.,  discussion  in  small  groups  to 
find  a  joint  solution  to  a  problem,  project-based  learning,  use  of  manipulatives),  rather 
than  conventional  (e.g.,  drill  and  skill  and  individual  worksheets).  In  other  areas,  fewer 
differences  were  found.  For  mstance,  both  NAS  and  non-NAS  teachers  reported  similar 
use  of  instructional  materials,  though  more  teachers  in  NAS  than  non-NAS  schools 
perceived  inadequate  materials  to  be  a  problem.  The  more  substantial  differences  we 
found  were  not  between  NAS  and  non-NAS  schools,  but  between  1998  and  1999,  which 
is  likely  a  reflection  of  the  dramatic  level  of  change  within  the  district  itself.  That  is, 
while  the  implementation  of  NAS  designs  was  not  high  relative  to  other  schools  and 
jurisdictions  (see  Berends  and  Kirby  et  al.,  2000;  Kirby  et  al.,  2001),  implementation  of 
NAS  designs  was  higher  in  1998  than  it  was  in  1999. 
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STUDENT  ACfflEVEMENT  IN  NAS  AND  NON-NAS  CLASSROOMS 

Because  instructional  conditions  varied  more  between  NAS  and  non-NAS  schools 
during  the  1997-1998  school  year,  and  we  wanted  to  examine  whether  such  variation  in 
instructional  conditions  was  related  to  student  achievement,  controlling  for  other 
student,  teacher,  classroom,  and  school  characteristics.  We  first  examined  relationships 
in  all  the  4*  grade  classrooms  in  the  district  and  then  in  the  sample  of  classrooms  for 
which  RAND  gathered  additional  survey  on  classroom  instruction  and  a  supplemental 
reading  test  (Stanford-9). 

As  expected  because  of  the  early  stages  of  implementation,  NAS  designs  had  no 
significant  effects  on  student  achievement.  More  importantly,  we  did  not  find  that 
instructional  conditions  promoted  by  reforms  as  such  as  NAS— including  teacher- 
reported  collaboration,  quality  of  professional  development,  and  reform-like 
instructional  practices— were  related  to  student  achievement  net  of  other  student  and 
classroom  conditions. 

However,  we  did  find  significant  effects  of  principal  leadership  on  the  TAAS 
reading  and  mathematics  scores.  Principal  leadership  in  our  analysis  was  measured  by 
teacher  reports  about  principals  who  clearly  communicated  what  was  expected  of 
teachers,  were  supportive  and  encouraging  of  staff,  obtained  resources  for  the  school, 
enforced  rules  for  student  conduct,  talked  with  teachers  regarding  instructional 
practices,  had  confidence  in  the  expertise  of  tire  teachers,  and  took  a  personal  interest  in 
the  professional  development  of  teachers.  Our  previous  analyses  have  shown  the 
importance  of  principal  leadership  in  implementing  the  designs  (Berends  and  Kirby  et 
al.,  2001;  Kirby  et  al.,  2001),  so  the  link  between  implementation  and  performance  is 
likely  to  be  important  as  more  and  more  schools  adopt  comprehensive  school  reform. 

POUCY  IMPLICATIONS 

Currently,  through  the  federal  funding  many  schools  throughout  the  coimtry  are 
attempting  NAS-like  reforms  using  the  federal  funding  provided  by  such  programs  as 
Title  I  and  the  Comprehensive  School  Reform  Demonstration  (CSRD)  program.  Our 
study  in  conjunction  with  the  other  RAND  studies  on  NAS  has  clear  implications. 
Schools  attempting  comprehensive  school  reforms  face  many  obstacles  during 
implementation,  and  because  of  this,  whole-school  designs  face  continuing  challenges  in 
significantly  raising  the  achievement  of  all  students.  This  is  particularly  important  to 
remember  when  setting  expectations  for  school  improvement  under  new  federal,  state. 
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and  local  programs— particularly  when  implementing  strategies  and  interventions  in 
high-poverty,  low-performing  settings. 

Because  the  target  of  the  federal  Title  I  and  CSRD  funds  is  primarily  high-poverty 
schools,  schools  most  likely  to  be  affected  by  the  CSRD  program  are  also  schools  that  are 
most  likely  to  face  very  fragmented  and  conflicting  environments,  difficult  and 
changing  political  currents,  new  accoimtability  systems,  trenchant  vmions,  serious  lack 
of  slack  resources  in  terms  of  teacher  time,  and  demoralized  teachers  given  the 
fluctuating  reform  agenda,  and  the  difficult  task  of  improving  student  performance 
xmder  these  types  of  conditions  (for  a  description  of  CSRD  schools  see  Kirby  and 
Berends  et  al.,  in  review). 

Given  this,  federal  and  state  policymakers  need  to  think  critically  about  their 
current  stance  of  simultaneously  promoting:  high  stakes  testing;  the  implementation  of 
comprehensive  school  reforms  that  promote  innovative  curriculum  and  instructional 
strategies;  and  the  implementation  of  multiple  other  concurrent  reforms.  The 
implementation  of  high  stakes  testing  regimes— the  apparent  outcome  of  many 
standards  based  reforms —might  preclude  the  adoption  of  rich  and  varied  curricula  that 
challenge  students  and  motivate  them  toward  more  in-depth  learning  experiences.  It 
certainly  prevents  it  when  other  more  "skill  and  drill"  like  reforms  are  mandated  on  top 
of  the  design-based  curriculum.  The  current  study  shows  that  high  stakes  tests  are  a 
two  edged  sword  in  this  environment.  On  the  one  hand,  high  stakes  tests  motivate 
schools  to  increase  performance  and  often  to  seek  out  new  curriculum  and  instructional 
strategies  associated  with  comprehensive  school  reforms.  On  the  other  hand,  those  very 
same  tests  provide  disincentives  to  adopt  richer,  more  in-depth  curriculum. 

Our  findings  are  consistent  with  Porter  and  Clrme's  scheme  for  better  educational 
policy  (see  Porter  et  al.,  1988;  Porter,  1994;  Clune,  1998).  They  posit  that  educational 
policies  such  as  comprehensive  school  reform  are  likely  to  influence  teachers  and 
students  to  the  extent  to  which  they  are  specific,  powerful,  authoritative,  consistent,  and 
stable.  Specificity  ox  depth,  is  the  extent  to  which  the  comprehensive  school  reform 
provides  detailed  guidance  or  materials  to  help  schools  and  teachers  understand  what 
they  are  supposed  to  do  (e.g.,  materials  that  describe  the  stages  of  implementing  the 
design  and  ongoing,  clear  assistance  strategies  to  further  promote  implementation). 
Power  refers  to  the  rewards  or  sanctions  attached  to  the  whole-school  reform,  such  as 
teachers  receiving  bonuses  or  greater  autonomy  if  they  comply  with  implementing  the 
design.  Authority  refers  to  the  degree  to  which  the  reform  policy  is  seen  as  legitimate 
and  as  having  the  support  oi  those  who  are  responsible  for  implementation.  If  respected 
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groups  or  policymakers  have  strong  positive  views  toward  whole-school  reform  and  if 
teachers  support  its  implementation,  the  design  is  likely  to  have  greater  mfluence  in 
changing  teaching  and  learning.  Consistency  or  alignment  refers  to  the  extent  to  which 
the  set  of  whole-school  interventions  and  strategies  are  aligned  with  a  common  mission 
and  vision,  both  within  the  school  and  the  district.  Stability  refers  to  the  reform  being 
sustained  over  time  in  a  coherent,  consistent  manner.  Policymakers  and  educators 
should  use  these  dimensions  as  a  means  for  thinking  critically  about  the  comprehensive 
school  reform  being  considered  cmd  whether  the  conditions  exist  for  it  to  flourish. 

Thinking  carefully  about  the  factors  necessary  to  promote  high-quality 
implementation  and  coherence  with  other  educational  policies  and  reforms  and 
ensuring  that  these  factors  are  present  and  aligned  in  schools  is  the  only  way  in  which 
comprehensive  school  reform  can  succeed  in  improving  the  learning  opportunities  of  all 
students,  particularly  those  in  high-poverty  settings. 
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GLOSSARY  OF  TERMS 


Symbol 

CSRD 

EDM 

ELOB 

IG 

ILT 

MRSH 

NAS 

NASDC 

RW 

SFA/RW 

SAISD 

SFA 

TAAS 

TEA 

TEKS 

TSSAS 

SALT 

Staiiford-9 


Definition _ 

Comprehensive  School  Reform  Demonstration 

Eveiyday  Math  of  the  University  of  Chicago 

Expeditionary  Leamin^Outward  Bound 

Instructional  Guide 

Instructional  Leadership  Teams 

Modem  Red  Schoolhouse 

New  American  Schools 

New  American  School  Development 

Corporation 

Roots  &  Wings 

Roots  &  Wings/Success  for  All 

San  Antonio  Independent  School  District 

Success  for  All 

Texas  Assessment  of  Academic  Skills 

Texas  Education  Agency 

Texas  Education  Knowledge  Standards 

Texas  Successful  Schools  Award  System 

San  Antonio  Leadership  Team 

Stanford  Open  Ended  Reading  Test,  Version  9 
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1.  NEW  AMERICAN  SCHOOLS'  AMBITIONS  FOR  CHANGING  HIGH  POVERTY 

CLASSROOMS 


New  American  Schools  (NAS),  a  private  non-profit  organization,  launched  its  efforts  for 
whole-school  reform  in  1991  to  address  the  common  perception  that  our  schools  were  failing 
students,  particularly  those  in  high-poverty  settings,  and  that  the  piecemeal  reform  efforts  had 
done  little  to  improve  the  nation's  educational  system. 

Based  on  the  premise  that  high-quality  schools  can  be  established  with  external 
providers  (design  teams)  supplying  assistance  to  schools  as  they  implement  whole-school 
models  of  reform,  NAS  set  out  to  help  schools  and  districts  significantly  raise  the  achievement 
of  large  numbers  of  students. 

SCALING-UP  NAS  DESIGN  TEAMS 

As  defined  by  NAS,  a  design  team  is  an  organization  that  provides  high-quality,  focused, 
ongoing  professional  development  for  teachers  and  administrators  organized  aroxmd  a 
meaningful  and  compelling  vision  of  what  students  should  know  and  be  able  to  do.  The 
vision,  or  design,  offers  schools  a  focus  for  their  improvement  efforts,  along  with  guidance  in 
identifying  what  students  need  to  know  and  be  able  to  do  and  how  to  get  there  (New 
American  Schools,  1997,  p.  6).  Glennan  (1998,  p.  11)  describes  a  design  further  saying  that  it 
"articulates  the  school's  vision,  mission,  and  goals;  guides  the  instructional  program  of  the 
school;  shapes  the  selection  and  socialization  of  the  staff;  and  establishes  common  expectations 
for  performance,  behavior,  and  accoimtability  among  students,  teachers,  and  parents." 

The  intent  of  NAS's  development  of  designs  was  to  "design  an  educational  environment 
to  bring  every  child  in  this  community  up  to  world  class  standards  in  English,  mathematics, 
science,  history,  and  geography,  prepared  for  responsible  citizenship,  further  learning,  and 
productive  emplo5anent.  No  question  about  schooling  should  be  off-limits;  no  answer 
assmned."  Its  interest  was  "in  the  comprehensive  reformation  of  entire  schools  or  sets  of 
schools.  [NAS]  believes  that  all  aspects  of  the  school  need  to  be  integrated  into  a  coherent, 
high-performance  learning  environment,  a  new  American  school"  (New  American  Schools, 
1991,  p.  9). 

NAS  is  currently  in  the  scale-up  phase  of  its  effort.  Its  designs  are  being  widely  diffused 
in  partnering  jurisdictions  across  the  nation.  NAS's  strategy  for  scale-up  is  based  on  the  belief 
that  school  transformation  can  only  take  place  with  strong  district  support  At  the  beginning  of 
the  scale-up  phase  in  1995,  NAS  sought  to  partner  with  jurisdictions  that  would  commit  to  five- 
year  partnerships  with  it  and  the  design  teams  to  create  a  supportive  environment  for 
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schcx)lwide  reform.  NAS  partnered  with  ten  jurisdictions:  Cincinnati,  Ohio;  Miami-Dade 
County,  FL;  several  districts  in  Kentucky;  Maryland;  Memphis,  TN;  Pittsburgh,  PA; 
Philadelphia,  P A;  San  Antonio,  TX;  San  Diego,  CA;  and  several  districts  in  Washington  State. 
All  of  these  jurisdictions  insisted  that  the  participating  schools  meet  district  or  state  standards 
and  that  students  be  assessed  against  district  and  state  mandated  tests. 

As  NAS  entered  the  scale-up  phase,  seven  design  teams  were  involved: 

•  Audrey  Cohen  College  [AC]  (currently  renamed  Purpose-Centered  Education); 

•  Authentic  Teaching,  Learning,  and  Assessment  for  All  Students  [AT]; 

•  Co-NECT  Schools; 

•  Expeditionary  Learning  Outward  Bound  [ELOB]; 

•  Modem  Red  SchooUiouse  [MRSKfj; 

•  Natior\al  Alliance  for  Restmcturing  Education  [NARE]  (currently  renamed  America's 
Choice  Design  Network);  and 

•  Roots  &  Wings  [RW].  ^ 

While  each  design  has  imique  features,  die  designs  commonly  emphasize  school  change 
in  the  following  areas  (referred  to  as  elements):  organization  and  governance;  teacher 
professional  development;  content  and  performance  standards;  curriculum  and  instructional 
strategies;  and  parent  and  community  involvement.  The  focus  in  this  analysis  is  whether 
designs  promote  classroom  learning  environments  that  affect  student  achievement 

Since  RAND's  programs  of  studies  began  at  the  beginning  of  scale-up,  NAS's  portfolio  of 
teams  has  changed  significantly.  For  instance,  another  design  -  Urban  Learning  Centers  (ULC) 
-  was  implementing  in  the  Los  Angeles  area  during  the  development  phase  of  NAS,  but  when 
scale-up  began,  this  design  team  was  not  included  in  the  NAS  portfolio  because  the  team  had 
not  shown  the  capacity  to  go  to  scale.  Since  that  time,  ULC  has  shown  this  capacity  and  is 
currently  being  marketed  by  NAS  as  one  of  its  designs.  Moreover,  Audrey  Cohen  (AC)  did  not 
show  the  progress  during  scale-up  that  NAS  desired,  and  this  team  was  dropped  from  the 
portfolio.  To  date,  NAS  has  10  teams  in  it's  portfolio:  Accelerated  Schools,  America's  Choice, 
ATLAS  Communities,  Co-NECT,  ELOB,  Leonard  Bernstein  Center,  Modem  Red  Schoolhouse, 
Success  For  All/Roots  &  Wings,  Turning  Points,  and  Urban  Learning  Centers.  Readers  can 
find  information  on  these  design  teams  at  http:/ / www.newamericanschools.org/teams. 

2  In  the  past  couple  of  years,  NAS  added  Success  for  AU  schools  to  its  portfolio  and  total  counts  of  schools 
that  it  ever  assisted.  However,  while  SFA  has  been  around  for  a  couple  of  decades,  NAS  provided 
funding  to  the  Success  for  All  Formdation  to  develop  and  implement  Roots  &  Wings,  which  not  only 
includes  the  reading  program  of  SFA,  but  also  builds  in  other  curricular  programs  such  as  Math  Wings 
and  WorldLab. 
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NAS  currently  encourages  the  implementation  of  comprehensive  school  approaches  in 
over  3,000  schools  by  advocating  the  adoption  of  designs  and  assistance  from  NAS  design 
teams.  This  coimt  includes  all  schools  in  which  designs  were  ever  implemented  including 
schools  that  dropped  designs  or  changed  to  a  different  design.  For  example,  with  the  transition 
of  National  Alliance  to  America's  Choice,  several  schools  in  Kentucky  dropped  the  NARE 
design  and  did  not  switch  to  America's  Choice.  These  schools  are  still  included  in  the  total 
coimt.  Moreover,  NAS  recently  expanded  to  ten  teams.  For  example,  by  adding  Accelerated 
Schools  to  its  portfolio  it  added  roughly  700  schools.  Moreover,  a  few  years  ago,  NAS  counted 
all  schools  implementing  the  Success  for  All  (SFA)  reading  program  in  its  portfolio,  even 
though  the  original  intent  of  NAS  was  to  expand  the  SFA  reading  program  into  other  subject 
areas  such  as  mathematics  and  social  studies,  thus  becoming  Roots  and  Wings. 

RELATIONSHIP  OF  NAS  TO  FEDERAL  SUPPORT  FOR  SCHOOLWIDE  CHANGE 

The  purposes  and  approaches  of  NAS  and  its  design  teams  are  the  same  as  those  of 
"schoolwide"  Title  I  programs^  and  the  Comprehensive  School  Reform  Demonstration  (CSRD) 
program  also  known  as  Obey-Porter  (see  Kirby  and  Berends  et  al.,  in  review).^  These  two 
programs  are  targeted  to  improve  the  performance  of  high-poverty  schools.  Each  intends  to 
improve  the  performance  of  at-risk  students  and  schools  by  having  schools  adopt  a  imified, 
coherent  approach  to  reform  rather  than  adding  fragmented  programs  or  investing  in 
personnel  dedicated  to  a  small  group  of  students  in  pull-out  programs.  Each  model  intends  to 
serve  all  students,  not  just  sub-groups  of  students.  Given  the  similar  intentions  of  NAS  and 
both  federal  initiatives,  findings  on  its  attempts  at  whole-school  change  can  help  inform  the 

^  "Schoolwide"  programs,  available  for  funding  since  1988,  aUow  schools  to  use  Title  1  money  with  other 
dollars  to  improve  sAool  performemce  as  opposed  to  targeting  Title  I  money  solely  to  qualified  students. 
The  1994  Improving  America's  Schools  Act  encourages  more  wide-range  adoption  of  schoolwide 
programs  (see  American  Association  of  School  Administrators,  1995; 

httfx/  /Vvww^ed.gov/Iegislation/ESEA).  Currently,  schools  can  use  their  Title  I  funding  to  improve  the 
entire  instructional  program  throughout  the  school  if  at  least  50  percent  of  the  students  within  the  school 
are  from  poor  families.  (For  a  discussion  of  the  1994  Improving  America's  Schools  Act  see  U.S. 
Department  of  Education,  1993;  and  Borman  et  al.,  1996). 

^  To  further  the  implementation  of  comprehensive,  whole-school  reforms,  the  Comprehensive  School 
Reform  Demonstration  program  (CSRD),  also  known  as  Obey-Porter,  was  established  in  November  1997. 
These  appropriations  committed  $145  miUion  to  be  used  to  help  schools  develop  comprehensive  school 
reform  based  on  reliable  research  and  effective  practices.  The  majority  (83  percent  in  FY98  and  77  percent 
in  FY99)  of  the  funds  are  committed  to  Title  I  schools.  Part  of  the  money  ($25  million  in  FY98  and  FY99) 
was  available  to  aU  public  schools,  including  those  ineligible  for  Tide  I,  as  part  of  the  Fund  for  the 
Improvement  in  Education  (FIE)  program.  Approximately  1,800  schools  will  receive  at  least  $50,000  per 
year  for  three  years  under  the  CSRD  program,  beginning  in  FY98.  There  was  an  increase  of  $75  miUion 
for  FYOO  ($50  million  in  Tide  I/Section  1502  funds  and  $25  miOion  in  FIE  funds)  over  the  $145  million 
appropriated  for  FY98  and  FY99,  which  will  allow  1,000  additional  schools  to  undertake  comprehensive 
reform  (see  Kirby  and  Berends  et  al.,  in  review;  http:  /  / www.ed.gov/offices/ QESE/compreform). 
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needed  policy  improvement  for  the  many  schools  that  serve  low-income  students  tiirough  the 
Title  1  and  the  CSRD  programs. 

PURPOSE  AND  STUDY  QUESTIONS 

This  study  aims  to  shed  light  on  the  policy  approach  and  strategies  of  the  New 
American  Schools'  (NAS)  educational  reform  initiative  that  incorporates  several  different 
whole-school  reform  designs.  Our  research  focuses  on  the  designs  fliat  were  implemented  in 
San  Antonio,  Texas,  a  district  that  was  viewed  during  the  course  of  our  data  collection  as 
having  a  supportive  environment  for  NAS  in  terms  of  educational  mission  and  vision,  finance 
and  governance  arrangements,  and  extant  policies.^  This  study  focuses  the  conditions  at  the 
district,  school,  and  classroom  level  that  are  related  to  design  implementation,  changes  in 
classroom  instruction,  and  student  achievement.  The  analyses  here  offer  both  useful  and 
provocative  insights  that  can  help  inform  tihe  NAS  effort  and  larger  federal  efforts  to  implement 
comprehensive  school  reform  aimed  at  improving  teaching  and  learning,  particularly  in  high- 
poverty  settings. 

Specifically,  this  study  addresses  the  following  questions: 

•  Do  the  NAS  designs  extend  beyond  changes  in  school  organization  and  governance 
and  permeate  classrooms?  Do  NAS  teachers  and  students  interact  with  each  other 
and  subject  materials  in  ways  that  reflect  the  innovative  curricular  and  instructional 
approaches  of  the  design  teams? 

•  What  factors  at  the  district,  school,  and  classroom  level  are  related  to 
implementation  of  designs  and  changes  in  classroom  instruction? 

•  Do  the  NAS  designs  and  the  design-based  assistance  provided  lead  to  changes  in 
curriculiun  and  instruction  and  thereby  achievement? 

STUDY  DESIGN 

During  the  course  of  this  study,  four  NAS  designs  were  being  implemented  in  San 
Antonio  -  Co-NECT,  Expeditionary  Learning/ Outward  Bound  (ELOB),  Modem  Red 
Schoolhouse  (MRSH),  and  Roots  and  Wings/Success  for  All  (SFA/RW).  In  order  to 
understand  how  the  NAS  design  teams  influenced  teaching  and  learning  across  subject  areas, 
we  monitored  a  sample  of  grade  teachers  across  these  four  designs  in  addition  to  similar 
teachers  in  non-NAS  schools  by  gathering  a  variety  of  data  about  classrooms.  In  cooperation 
with  the  San  Antonio  school  district,  we  selected  our  sample  of  teachers  from  New  American 

^  While  there  are  several  school  districts  in  the  city  of  San  Antonio,  throughout  our  study  we  refer  to  the 
schools  within  the  San  Antonio  Independent  School  District  as  San  Antonio  schools. 
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Schools'  sites  and  some  comparison  sites,  not  implementing  any  whole-school  reform  as  yet.^ 
Because  San  Antonio  did  not  want  to  overburden  its  schools  by  mandating  their  participation 
in  research  activities,  the  Associate  Superintendent  of  Curriculum  and  Instruction  sent  a  letter 
to  all  65  elementary  schools  in  the  district  requesting  tiieir  participation  in  the  RAND  study. 
While  not  a  random  sampling  process,  which  is  not  feasible  given  the  self-selection  of  designs 
by  schools  and  the  limited  number  of  schools  implementing  designs,  district  staff  assisted 
RAND  in  selecting  teachers  that  were  representative  of  the  elementary  schools  in  the  district 
(e.g.,  representative  in  terms  of  academic  performance  and  school  demographic  characteristics). 

We  also  relied  on  information  from  surveys  of  teachers  and  principals,  student 
achievement  and  backgroimd  characteristics,  observations  and  classroom  artifacts,  and 
interviews  with  district  staff  to  help  us  address  the  questions  of  this  study. 

UNDERSTANDING  THE  RELATIONSHIPS  AMONG  NAS  DESIGNS,  CLASSROOM 
INSTRUCTION,  AND  STUDENT  ACHIEVEMENT 

While  NAS  aims  to  help  schools  and  districts  significantly  raise  the  achievement  of  large 
numbers  of  students  through  design-based  assistance,  improving  student  and  school 
performance  is  a  critical  goal  of  all  comprehensive  school  reforms. 

To  understand  flie  relative  impact  of  NAS  designs  (and  comprehensive  school  reform 
models),  we  need  to  incorporate  the  perspectives  of  a  variety  of  actors  throughout  the  system. 
No  doubt,  the  process  of  school  change  to  improve  student  achievement  is  complex  and 
difficult.  It  requires  the  coordination  of  a  variety  of  actors  and  factors  to  make  it  work.  The 
framework  portrayed  in  Figure  1.1  portrays  some  of  that  complexity.  In  our  other  research  on 
NAS,  we  have  highlighted  those  conditions  that  are  related  to  implementation  of  NAS  designs 
at  the  school  level  (Bodilly,  1998, 2001;  Berends,  2000;  Berends  and  Kirby  et  al.,  2001;  Kirby, 
Berends,  and  Naftel,  forthcoming).  In  this  report,  we  focus  on  those  factors  at  the  district, 
school,  classroom,  and  student  levels  that  are  related  to  changes  in  classroom  instruction  and 
student  achievement  scores. 

The  approach  of  NAS  and  its  design  teams  is  the  designs  and  the  assistance  they  provide 
to  schools  and  teachers  will  promote  clianges  in  classroom  instruction  that  lead  to  increases  in 
student  learning.  Even  after  controlling  for  other  important  contexts— e.g.,  student,  teacher, 
school,  and  district— these  relationships  among  designs,  classroom  instruction,  and 
achievement  are  critical  for  improving  schools,  classrooms,  and  student  learning:  a  critical 

^  At  the  time  of  this  study,  the  San  Antonio  superintendent  had  a  stated  goal  of  all  the  schools  in  the 
district  adopting  some  type  of  whole-school  reform  design  (NAS  or  non-NAS  designs)  by  fell  1999. 
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assumption  of  NAS  itself.  Certainly,  the  relationships  in  Figure  1.1  can  be  portrayed  in  much 
more  complex 


Figure  2.1— A  Conceptual  Framework  for  Understanding  the  Relationships  Among  NAS 
Designs,  Classroom  Instruction,  and  Student  Achievement 

manner  with  additional  arrows  in  multiple  directions,  suggesting  a  myriad  of  direct  and 
indirect  effects.  However,  tire  focus  of  this  report  is  on  tirose  classroom  practices  promoted  by 
NAS  designs  and  influenced  by  factors  at  the  district  and  school  levels.  Figure  1.1  is  intended 
to  emphasize  those  relationships,  some  of  which  we  highlight  further  in  the  sections  that 
follow. 

Core  Elements  of  Designs 

To  accomplish  the  goal  of  improving  performance,  each  design  team  has  a  "theory  of 
action"  that  establishes  a  link  between  elements  of  the  design  (which  include  curriculiun  and 
mstruction,  professional  development,  school  governance),  classroom  learning  environments. 
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and  student  performance.  The  NAS  designs  range  from  relatively  specific  descriptions  of  how 
schools  should  be  organized  and  what  materials  and  professional  development  should  be 
relied  on  to  less  specific  visions  and  processes  for  school  restructuring. 

One  of  the  more  specific  NAS  designs  is  SFA/RW,  which  builds  on  years  of  research  and 
implementation  experiences  wifh  the  reading  and  writing  program  Success  for  All.  SFA/RW 
provides  an  abxmdance  of  print  materials,  assessments,  professional  development,  and 
specified  organizational  changes  (e.g.,  homogeneous  instructional  groups  that  are  reorganized 
frequently  to  address  students'  needs).  The  design  begins  implementation  with  a  specific  focus 
on  changing  curriculum  and  instruction. 

In  contrast,  some  of  the  other  NAS  designs  are  more  process  oriented.  For  instance, 

ELOB  is  less  structured  than  SFA/RW  and  is  based  on  design  principles  that  reflect  the  designs 
origins  in  the  Outward  Bound  program.  Students'  experiences  in  ELOB  schools  consist 
primarily  of  engaging  in  multi-disciplinary,  project-based  learning  expeditions  that  include 
intellectual,  service,  and  physical  dimensions.  Teachers  play  a  critical  role  in  developing  the 
expeditions,  which  involve  a  great  deal  of  effort  and  imagination. 

Thus,  it  is  important  to  remember  the  unique  attributes  of  each  design  in  terms  of  its 
complexity  and  specificity,  elements  of  schooling  emphasized,  and  tire  different  strategies  for 
implementation.  Certainly,  we  cannot  capture  all  of  the  uniqueness  of  each  design  in  the 
current  analysis.  For  further  elaboration,  we  suggest  looking  at  RAND's  other  NAS  studies  that 
have  pointed  to  these  characteristics  and  the  importance  of  looking  at  changes  in  designs  over 
time  (see  Bodilly,  2001;  Berends  and  Kirby  et  al.,  2001;  Berends,  1999, 2000;  Gleiman,  1998). 

Student  Characteristics 

An  important  set  of  factors  that  affect  implementation  of  school  restructuring  efforts 
and  their  effects  is  student  backgroimd  (e.g.,  socioeconomic  status  or  poverty  level,  race- 
ethnicity,  gender,  English  language  proficiency,  and  mobility)  and  prior  achievement.  Within 
school  reform  efforts,  it  is  important  to  imderstand  how  changes  in  schooling  activities  are 
related  to  students'  social  background  characteristics,  their  home  enviromnents,  mobility 
patterns  between  schools,  and  their  preexisting  levels  of  academic  achievement  in  school 
(Berends  et  al.,  1999;  Koretz,  1996;  Meyer,  1996).  Policymakers  focus  on  manipulating  the 
"lever"  at  the  school  level  to  improve  learning  opportunities  and  performance.  Yet,  when 
assessing  the  impact  of  NAS  and  its  designs  on  classroom  instruction,  it  is  important  to 
understand  the  net  influence  of  school  reform  activities,  especially  with  ffie  number  of  studies 
ffiat  have  shown  the  importance  of  student  background  in  the  learning  process  (see  Coleman  et 
al.,  1966;  Jencks  at  al.  1972;  Gamoran,  1987, 1992;  Bryk  et  al.,  1993).  That  is,  when  assessing  the 
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effects  of  classroom  environments,  we  need  to  control  for  other  important  school,  classroom, 
and  student  factors. 

Teacher  Characteristics 

Without  willing  and  able  teachers  who  embrace  reform  and  provide  the  necessary 
leadership  to  imdertake  it,  no  change  can  be  enacted,  no  matter  how  effective  it  may  be. 
Teachers  are  the  "street  level  bureaucrats"  at  the  core  of  educational  change  (Weatherly  and 
Lipsky,  1977)  and  as  Fullan  succinctly  stated,  educational  reform  depends  on  "what  teachers  do 
and  think-it's  as  simple  and  as  complex  as  thaT'  (Fullan,  1991,  p.  117). 

Educators  must  respond  to  multiple,  simultaneous  pressures  and  demands.  For  many 
teachers,  policy  goals  and  activities  are  simply  part  of  a  broader  environment  that  presses  in 
upon  their  classrooms.  Their  ability  to  cope  with  these  demands,  and  their  commitment  to 
change  are  crucial  to  coherent  and  sustained  implementation.  Thus,  teachers  matter:  their 
experience,  subject-based  expertise,  attitudes,  and  orientations  are  important  in  determining  the 
degree  and  level  of  implementation. 

In  short,  teachers  are  central  to  all  organizational  changes  that  alter  student-teacher 
interactions  occurring  in  classrooms  to  improve  student  learning  (Gamoran  et  al.,  1995;  Oakes 
et  al.,  1992).  Over  time,  teachers  carry  with  them  a  great  deal  of  knowledge  based  on  their 
educational  attainment,  teaching  experience,  and  other  personal  characteristics  that  together  are 
likely  related  to  their  engagement  in  schoolwide  restructuring  activities  (Louis  and  Marks, 

1998).  Thus,  it  is  important  to  examine  the  relationships  among  various  teacher  background 
characteristics,  classroom  instruction,  and  student  achievement 

School  and  Classroom  Characteristics 

Characteristics  of  schools  and  classrooms  are  also  likely  to  influence  the  adoption  of 
schoolwide  designs  and  their  effects  on  classrooms  and  students.  In  our  work,  we  examine 
whether  school  and  classroom  characteristics  such  as  the  minority  and  poverty  composition  are 
related  to  implementation  and  performance  (Berends  and  Kirby  et  al.,  2001;  Berends,  1999, 

2000).  Schools  and  classrooms  that  face  economic  challenges,  which  are  often  highly  correlated 
with  a  disproportionate  number  of  minority  students,  may  experience  difficulty  engaging  in 
restructuring  efforts  such  as  whole-school  designs  (Kirby  and  Berends  et  al.,  2001).  Yet, 
because  federal  funding  such  as  Title  I  is  oriented  towards  disadvantaged  students  and  schools, 
the  effects  of  socioeconomic  and  minority  composition  are  likely  to  be  mediated.  In  fact,  since 
the  1994  reauthorization  of  Title  I,  schools  with  more  than  half  of  their  students  eligible  for  free 
and  reduced  priced  lunch  may  use  Title  I  funds  for  schoolwide  programs.  Thus,  there  may  be  a 
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positive  relationship  between  high-poverty  schools  and  the  implementation  of  schoolwide 
reform  models  such  as  NAS  designs  because  of  such  funding  sources. 

DistricVState  Context 

Research  also  underscores  the  importance  of  the  external  environment  in  the  process  of 
change,  especially  district  and  state  support  and  stability  of  leadership  (BodiUy  and  Berends, 
1999;  BodiUy,  1998;  Glennan,  1998;  Fullan,  1991).  Districts  and  states  can  facilitate  and  foster 
change  by  providing  resources  for  the  school  and  for  professional  staff  development,  and  by 
showing  active  support  for  schools  implementing  designs. 

The  federal  and  state  policy  context  is  likely  to  play  a  role  in  implementing  schoolwide 
reform  (see  FuUan,  1991, 2001;  Koretz  and  Barron,  1998).  For  example,  the  recent 
Comprehensive  School  Reform  Demonstration  (CSRD)  program  directly  supports  design-based 
reforms  such  as  NAS  by  providing  at  least  $50,000  to  schools  to  pay  for  the  related  services. 
Some  states  and  districts  with  high-stakes  accountability  systems  may  force  low-performing 
schools  to  adopt  designs.  They  may  also  facilitate  a  more  effective  matching  process  for  schools 
to  select  designs  based  on  their  local  needs  (see  BodiUy  and  Berends,  1999;  BodiUy,  1998;  Smith 
etal.,  1996). 

BodUly  (1998)  foimd  that  districts  play  a  strong  role  in  determining  the  initial  and 
sustained  viabiUty  of  the  relationship  between  the  school  and  the  design  team.  Early  on  in  the 
scale-up  phase,  many  schools'  staff  members  complain  about  the  district' s  poor  planning  and 
scanty  provision  of  time  to  make  decisions,  issues  brought  up  in  other  assessments  of  the 
adoption  of  schoolwide  programs  (Wong  and  Meyer,  1998). 

RAND's  prior  case  studies  (see  BodUly,  1998;  BodiUy  and  Berends,  1999)  reveal  that 
higher  average  levels  of  implementation  are  foimd  in  districts  that  have  a  stable  district 
leadership,  places  a  high  priority  on  the  effort,  lack  a  major  budget  crisis  or  other  crisis,  and 
have  a  history  of  trust  between  the  central  office  and  the  schools.  School-level  respondents 
directly  link  these  factors  to  greater  efforts  at  implementation.  When  these  factors  are  missing, 
school  respondents  report  that  their  own  efforts  staU  or  are  less  intense. 

WhUe  the  political  support  of  the  district  is  crucial  for  design  implementation,  its 
attention  can  be  buttressed  by  significant  changes  in  regulatory  and  financial  practices.  Schools 
attempting  comprehensive  school  reform  to  address  their  particular  problems  can  be  supported 
through  increased  site-level  control  over  their  curriculum  and  instruction,  their  budgets,  their 
positions  and  staffing,  and  most  essentiaUy  their  mission.  Comprehensive  school  reform  is  not 
confined  to  the  adoption  of  a  new  mandated  curriculum  or  a  few  new  instructional  strategies. 
Instead,  it  may  require  rethinking  and  adoption  at  the  school-level  of  new  curriculum  and 
instructional  approaches  and  tiie  accompanying  professional  development.  District  flexibUity 
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in  allowing  schools  to  pursue  this  rethinking  is  a  critical  aspect  for  design-based  schools. 
Development  and  implementation  of  such  curriculum  and  instructional  strategies  at  the  school 
level  may  be  significantly  hampered  without  district  support  though  resource  allocation  for 
instructional  positions,  materials,  technology,  professional  development,  etc. 

In  short,  the  district-level  politics,  policies,  and  practices  may  promote  or  derail  the 
schoolwide  reform  efforts  such  as  the  NAS  designs.  In  fact,  schools  may  look  to  district 
leadership,  climate,  and  regulations  to  imderstand  if  it  is  worth  their  time  and  effort  to  invest  in 
transforming. 

External  Assistance  by  Design  Teams 

How  schools  go  about  selecting  a  design  has  implications  for  the  implementation  that 
follows  (Bodilly,  1998;  Ross  et  al.,  1997).  For  example,  if  a  school  is  forced  to  adopt  a  design,  it 
is  not  surprising  that  teachers  resist  engaging  in  its  activities.  Yet,  some  schools  are  often 
targets  for  forced  restructuring  efforts,  particularly  those  that  exhibit  chronic  poor  performance. 
Thus,  a  critical  aim  of  the  NAS  designs  before  implementation  even  begins  is  to  obtain  the  buy- 
in  of  teachers  for  the  planned  restructuring  activities.  Most  of  the  designs  require  between  75- 
80  percent  of  the  teachers  voting  in  favor  of  the  designs.  The  rationale  is  tiiat  if  the  vast 
majority  of  the  staff  votes  to  adopt  the  design,  tiiey  will  commit  to  making  the  changes 
necessary  during  the  implementation  process. 

Clear  communication  by  designs  to  schools  is  critical  for  not  only  the  selection  of  any  one 
design,  but  also  the  implementation  of  it  -  something  that  external  assistance  providers  have 
fotmd  challenging  when  attempting  to  help  a  large  number  of  schools  (Bodilly,  1998). 
Communication  to  schools  both  during  the  selection  and  implementation  process  can  take 
several  different  forms,  including  design  fairs,  print  materials,  use  of  computer  software  and 
the  internet,  workshops,  retreats,  school  visits,  and  site-based  facilitators.  For  instance,  school 
visits  by  design  team  staff  on  a  regular  basis  to  help  teachers  address  issues  related  to 
developing  curriculiun  imits  or  the  use  of  rubrics  to  assess  students  are  intended  to  help 
teachers  implement  project  based  learning  and  the  assessment  of  that  learning  within  the 
context  of  the  design.  Other  types  of  communication  might  be  effective  as  well,  and  the  clearer 
and  more  consistent  the  information  provided  about  implementation  by  designs  to  schools,  the 
smoother  the  implementation  process  is  likely  to  be. 

For  implementation  of  any  program,  resources  are  critical  (Keltner,  1998;  McLaughlin, 
1990).  It  is  a  common  finding  that  when  resources  decrease  or  disappear,  the  implementation  is 
likely  to  diminish  (Glennan,  1998;  Montjoy  and  O'Toole,  1979).  If  teachers  receive  needed 
funds,  professional  development  from  design  teams  for  design  implementation,  materials  to 
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support  implementation,  and  the  time  to  plan  and  develop  their  programs,  it  is  likely  that 
implementation  will  deepen  over  time. 

By  gathering  information  about  different  aspects  of  the  system  from  the  district  to 
teachers  and  students  in  schools,  we  hope  to  shed  light  on  how  the  NAS  initiative  is  unfolding 
in  terms  of  implementation  and  outcomes  in  a  high-poverty,  low-performing  district  that  was 
inviting  and  provided  significant  resources  to  schools  of  implementing  NAS  designs.  We  turn 
to  this  rmderstanding  in  the  chapters  that  follow. 

ORGANIZATION  OF  REPORT 

Chapter  Two  provides  details  of  die  RAND  sample  of  NAS  schools  and  classrooms 
analyzed  for  this  study,  and  presents  a  profile  of  these  schools  and  classrooms  in  terms  of  their 
student  and  teacher  demographics.  Chapter  Three  places  NAS  design  implementation  within 
the  context  of  a  high-poverty  district  We  describe  various  aspects  of  the  district  environment 
in  which  designs  were  implemented  and  discuss  the  implications  for  comprehensive  school 
reform. 

Chapter  Four  presents  results  from  a  variety  of  data  sources  (e.g.,  teacher  surveys, 
classroom  observations,  interviews,  and  classroom  artifacts)  that  describe  school  organization, 
adoption  and  support  of  the  NAS  designs,  and  teachers'  training  and  professional  development 
in  NAS  and  non-NAS  schools.  In  Chapter  Five,  we  draw  on  similar  data  to  describe  ffie 
instructional  activities  in  NAS  and  non-NAS  classrooms.  Chapter  Six  examines  the  relationship 
between  instructional  activities  and  student  achievement,  controlling  for  other  factors.  Finally, 
Chapter  Seven  discusses  the  policy  implications  of  our  results. 
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2.  SOURCES  OF  DATA 


Fifteen  public  school  districts  serve  more  than  200,000  students  living  in  the  city  of  San 
Antonio  and  surroimding  communities.  The  San  Antonio  Independent  School  District  (SAISD) 
is  the  second  largest  district  in  the  coimty  and  the  seventh  largest  in  the  state.  Most  of  the 
approximately  55,000  students  who  attend  schools  in  San  Antonio  live  within  the  city  limits, 
and  the  district  has  the  highest  proportion  of  students  who  are  eligible  for  free  and  reduced 
lunch  in  the  county.^  Nearly  95  percent  of  students  in  San  Antonio  are  African  American  or 
Hispanic.  Since  1994,  the  proportion  of  San  Antonio  students  who  do  not  earn  passing  rates  on 
the  Texas  Assessment  of  Academic  Skills  (TAAS)  in  each  school  year  has  consistently  been  tiie 
lowest  or  second  lowest  in  the  coimty. 

It  is  within  this  context  of  high-poverty  and  low  student  performance  that  elementary 
schools  in  San  Antonio  began  the  process  of  adopting  New  American  Schools  (NAS)  reform 
models.  Three  schools  in  die  district  began  implementation  during  the  1995-1996  school  year, 
nine  schools  in  the  following  year,  twenty  schools  were  added  during  the  1997-1998  school  year 
and  by  the  1998-1999  school  year  thirty-nine  of  sixty-four  elementary  schools  in  the  district 
were  NAS  schools.  The  number  of  schools  adopting  specific  designs  in  each  year  is  listed  in 
Table  2.1. 


Table  2.1 

School  Adopting  NAS  Designs  in  San  Antonio  by  Year 


Number  of  Schook 

1995-1996 

1996-1997 

1997-1998 

1998-1999 

Totals 

Co-NECT 

3 

1 

4 

ELOB 

1 

2 

3 

MRSH 

5 

4 

1 

10 

SFA/RW 

2 

2 

13 

5 

22 

Non-NAS 

25 

Total 

64 

py 

For  more  information  see  http:/ /www.saisd.net  and  Texas  Education  Agency  web  page 
http:/  /wv\m-.tea.state.tx. 
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RAND  collected  data  on  a  sample  of  4*  grade  teachers  and  flieir  students  during  two 
school  years,  1997-1998  and  1998-1999  (see  Tables  2.2  and  2.3).  Generally,  in  each  school  year 
we  were  able  to  gather  teacher  survey  data  and  supplemental  student  test  scores  in  reading 
(Stanford-9),  including  over  850  students  in  over  60  classrooms  in  over  20  schools.  Moreover, 
during  the  course  of  this  study,  we  were  able  to  obtain  information  on  all  the  teachers  and 
students  in  the  district  to  provide  a  benchmark  for  the  analyses  reported  here.  In  1997-1998,  we 
were  also  able  to  observe  and  gather  classroom  artifacts  from  12  teachers  in  NAS  and  non-NAS 
schools,  and  in  the  following  year,  we  were  able  to  gather  such  data  from  about  19  teachers. 
Each  of  these  data  collection  efforts  is  described  more  fuUy  in  the  sections  that  follow. 

Table  2.2 

Target  Sample  of  Schools  Compared  with  Final  Study  Sample 
by  Type  of  Data  Collection  and  NAS  Design  Team 


Number  of  Schools  in  1997-1998  School  Year 


Requested  to 
Participate 

Returned 

Teacher 

Surveys 

Returned 

Principal 

Surveys 

Returned 

StanfQrd-9 

Testing 

Classroom 

Observations 

Co-NECT 

2 

2 

2 

2 

1 

ELOB 

2 

2 

1 

1 

1 

MRSH 

4 

4 

4 

4 

2 

SFA/RW 

8 

8 

8 

9 

1 

Non-NAS 

10 

8 

9 

10 

2 

Total 

26 

24 

24 

26 

7 

Number  of  Schools  in  1998-1999  School  Year 


Requested  to 
Participate 

Returned 

Teacher 

Surveys 

Returned 

Principal 

Surveys 

Returned 

Stanford-9 

Testing 

Classroom 

Observations 

2 

2 

2 

2 

2 

ELOB 

2 

2 

1 

2 

2 

MRSH 

4 

4 

2 

4 

2 

SFA/RW 

8 

8 

7 

8 

2 

Non-NAS 

7 

7 

8 

7 

2 

Total 

23 

23 

20 

23 

10 

Total 


23 


23 


10 
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Table  2.3 

Target  Sample  of  Teachers  Compared  with  Final  Study  Sample 
by  T3rpe  of  Data  Collection  and  NAS  Design  Team 


Number  of  Teachers  in  1997-1998  School  Year 


Requested  to 
Participate 

Returned 

Teacher 

Surveys 

Returned 
Surveys  & 
Stanford-9 
Testing 

Observations 

Co-NECT 

6 

6 

6 

3 

ELOB 

4 

4 

2 

1 

MESH 

12 

10 

4 

SFA/RW 

26 

23 

22 

2 

Non-NAS 

26 

23 

23 

2 

Total 

74 

66 

63 

12  1 

Number  of  Teachers  in  1998-1999  School  Year 

Requested  to 
Participate 

Returned 

Teacher 

Surveys 

Returned 
Surveys  & 
Stanford-9 
Testing 

Observations 

Co-NECT 

11 

11 

10 

4 

ELOB 

8 

8 

6 

5 

MRSH 

13 

13 

11 

3 

SFA/RW 

32 

32 

27 

4 

Non-NAS 


Total 
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While  the  RAND  sample  of  NAS  and  non-NAS  schools  cannot  be  considered  random, 
district  staff  indicated  that  the  schools  selected  were  typical  of  elementary  schools  in  the 
district.  Comparisons  of  demographic  and  other  characteristics  for  students  (i.e.,  gender,  race. 
Limited  English  Proficient  [LEP],  special  education  status,  average  test  scores,  and  mobility 
rates)  and  teachers  (i.e.,  gender,  race,  highest  degree  earned,  years  of  teaching  experience) 
indicated  no  significant  differences,  on  average,  between  the  RAND  sample  and  district 
populations.  Each  teacher  selected  was  asked  to  administer  the  Stanford-9  to  his  or  her  4* 
grade  students  and  to  complete  a  teacher  survey.  Teacher  focus  groups  were  conducted  in 
eight  schools.  A  subset  of  teachers  agreed  to  provide  classroom  logs,  samples  of  student  work 
and  allowed  classroom  observations  once  in  the  spring  of  the  1997-1998  school  year  and  three 
to  four  times  in  the  1998-1999  school  year.  In  addition,  principals  in  the  sample  schools  were 
asked  to  complete  a  telephone  interview,  during  which  a  survey  was  completed.  A  brief 
description  of  these  data  collection  efforts  follows,  and  these  are  summarized  in  Table  2.4. 

TEACHER  DATA 

In  the  late  spring  of  the  1997-1998  school  year,  with  the  help  of  district  staff,  we 
contacted  74  teachers  in  26  schools  to  participate  in  the  study.  Three  of  the  schools  refused  to 
participate  in  our  study.  Of  those  74  teachers  initially  contacted,  63  teachers  in  23  schools 
agreed  to  participate,  returned  completed  teacher  surveys,  and  their  students  completed  the 
Standford-9  reading  test  resulting  in  an  85  percent  response  rate  for  teachers  and  classes  with 
student  achievement  scores. 

In  1998-1999,  we  returned  to  the  23  schools  that  participated  in  our  study  the  previous 
year.  Because  we  wanted  to  increase  our  sample  of  teachers,  we  supplemented  our  teacher 
sample  and  contacted  83  teachers  in  diese  23  schools.  Of  those  contacted,  we  received 
completed  teacher  surveys  and  Stanford-9  tests  from  73  teachers  (88  percent).  Between  the 
spring  of  1998  and  1999,  one  of  our  sampled  schools  went  from  having  no  design  in  place  to 
adopting  SFA/RW. 
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Table  2.4 

RAND  Classroom  Study  in  San  Antonio  Data  Collection 


Type  of  Data _  Inf ormation  Provided 


Teacher  Survey 

• 

• 

• 

• 

• 

Design  team  program  characteristics 
Instructional  strategies 

Professional  development  activities 

Teacher  background 

Classroom  climate  and  other  characteristics 

Teacher  Logs  of  Classroom  Activities 

• 

• 

Design  team  program  characteristics 
Instructional  strategies 

Observations  of  Classroom  Instruction 

• 

• 

Design  team  program  characteristics 
Instructional  strategies 

Teacher  Interviews  &  Focus  Groups 

• 

• 

• 

• 

• 

• 

Design  team  program  characteristics 
Instructional  strategies 

Design  team  implementation  benchmarks 
Professional  development  activities 

Common  planning  time 

Resources  for  implementation 

Principal  &  Instructional  Guide  Interviews 

• 

• 

• 

• 

NAS  design  implementation 

School  chmate  and  other  characteristics 
Professional  development  activities 

Resources  for  implementation 

Design  Team  Interviews 

• 

• 

• 

• 

• 

• 

Design  team  program  characteristics 
Instructional  strategies 

Design  team  implementation  benchmarks 

NAS  design  implementation 

Professional  development  activities 

Resources  for  implementation 

District  Interviews 

• 

• 

• 

• 

NAS  design  implementation 

Professional  development  activities 

District  policies 

Resources  for  implementation 

Student  Characteristics  and  Performance 

• 

• 

• 

TAAS  mathematics,  reading,  and  writing 
scores  at  student  and  item-level  (linked  to 
teachers  and  schools) 

Stanford  Open-ended  Reading  Achievement 
Test  9*  Edition  at  student  and  item-level 
(linked  to  teachers  and  schools) 

Demographic  and  individual  characteristics 
of  students 
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Not  all  teachers  had  complete  survey  data  across  both  years,  because  of  different 
teachers  being  included  in  both  years.  Thus,  for  the  longitudinal  descriptions  of  classrooms 
implementing  NAS  and  non-NAS  designs,  we  track  indicators  for  40  of  teachers  for  whom  we 
have  complete  data  in  both  the  1997-1998  and  1998-1999  school  years  (see  Table  2.5).  In 
addition  to  these  teacher  data  from  RAND  surveys,  we  were  also  able  to  obtain  information  on 
teachers  from  the  district,  such  as  demographic  characteristics  (race-ethnicity  and  gender), 
years  of  experience,  and  highest  degree  obtained. 

Table  2.5 

Longitudinal  Sample  of  Teachers  in  NAS  and  Non-NAS  Schools, 
1997-1998  and  1998-1999  School  Yealrs 


Co-NECT 

ELOB 

MRSH 

SFA/RW 

Non-NAS 

Totals 

Number  of  Schools 

2 

2 

3 

6 

7 

20 

Number  of  Teachers 

4 

3 

8 

11 

14 

40 

*Teachers  who  completed  the  survey  in  both  spring  1998  and  spring  1999  and  who  were  in  the  same 
school,  same  design,  and  teaching  4**^  grade  in  both  years. 

We  discuss  the  results  from  this  longitudinal  sample  of  teachers  in  Chapter  Four  and 
Five.  We  Limit  our  discussion  to  this  longitudinal  sample  of  40  4^  grade  teachers  who 
completed  surveys  in  both  the  spring  of  1998  and  1999  and  remained  in  the  same 
school/design/ teaching  assignment.  We  do  this  because  we  are  interested  in  any  changes  that 
may  occur  during  the  early  stages  of  implementation  in  school  organization,  teachers' 
professional  work  lives,  and  their  classroom  instruction.  For  tfiis  reason,  the  surveys  asked  a 
broad  set  of  questions.  While  we  analyzed  the  larger  samples  in  both  years  to  check  our  results, 
we  do  not  report  on  the  larger  samples  because  they  too  are  convenience  samples  and  not 
random  samples. 

Because  of  the  small  sample  size  of  the  longitudinal  sample  analyzed,  we  do  not  focus 
much  attention  on  testing  the  statistically  significant  differences  between  NAS  and  non-NAS 
teachers.  Given  the  design,  most  standard  statistical  tests  comparing  the  40  NAS  and  non-NAS 
teachers  in  the  longitudinal  sample  would  fail  to  detect  many  real  differences.  However,  in 
conjunction  with  the  qualitative  data  from  this  study,  the  NAS-non-NAS  comparisons  shed 
light  on  a  variety  of  factors  related  to  implementing  NAS  designs  in  a  high-poverty  urban 
district. 
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Surveys 

The  teacher  survey  fielded  during  the  spring  1998  semester  and  then  again  in  spring  1999 
was  designed  to  provide  a  broad  measure  of  instructional  practices  in  NAS  and  non-NAS 
classrooms.  Teachers  were  asked  to  report  on  a  range  of  instructional  strategies,  some 
reflecting  a  focus  on  basic  skills  and  tend  toward  more  conventional  practices,  and  others 
reflecting  more  reform-like  practices.  The  NAS  designs  all  emphasize  promoting  changes  in 
instructional  conditions  whether  through  building  basic  skills  and  then  higher-order  thinking 
(e.g.,  SFA/RW)  or  through  theme-based  projects  that  last  for  several  weeks  (e.g.,  Co-NECT  or 
ELOB)  (see  Bodilly,  2001).  We  would  then  expect  implementation  of  designs  to  result  in 
changes  in  teaching  strategies. 

General  topics  covered  in  the  survey  include  school  and  classroom  characteristics, 
instructional  strategies  and  materials,  skills  and  assessments  emphasized,  resources,  parent 
involvement  and  community  relations,  impact  of  design  team  and  reform  efforts,  professional 
development,  and  perceptions  and  attitudes  towards  teaching. 

Two  versions  of  the  survey  were  fielded  in  each  year,  one  to  4*  grade  teachers  in  a 
sample  of  schools  adopting  NAS  designs,  the  other  to  4*  grade  teachers  in  non-NAS  schools. 
The  two  forms  of  the  surveys  varied  only  slightly.  For  instance,  three  items  specifically  related 
to  the  implementation  of  NAS  designs  was  not  included  in  the  survey  received  by  non-NAS 
teachers.  A  few  items  in  other  sections  also  referred  specifically  to  NAS  designs.  On  the  non- 
NAS  version,  these  items  were  either  omitted  or  had  slightly  different  wording  (i.e.,  whereas 
NAS  teachers  were  asked  about  the  NAS  design  being  implemented  in  their  school,  non-NAS 
teachers  were  asked  about  the  school  reform  efforts  in  their  district).  For  example,  an  item  on 
the  NAS  version  that  asked  if  an  activity  was  "specifically  oriented  toward  fire  design  team 
program  activities"  was  changed  to  "specifically  oriented  toward  fire  reform  efforts  of  San 
Antonio"  on  the  non-NAS  version. 
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Longitudinal  Sample  of  40  Teachers  Compared  with  Elementary  Teachers  in  District 

Overall,  it  appears  that  in  demographic  terms,  the  longitudinal  survey  sample  of  the  40 
teachers  is  a  fairly  representative  group  of  teachers  within  the  school  district.  There  were  few 
differences  when  comparing  teachers  in  our  sample  to  all  4*  grade  teachers  in  the  San  Antonio 
school  district  during  tire  1997-1998  (Table  2.6).  Teachers  in  this  sample  and  the  district  as  a 
whole  were  similar  when  considering  their  gender  or  racial-ethnic  characteristics  or  average 
years  of  experience.  Whereas  40  percent  of  teachers  in  the  district  had  earned  master' s  degrees, 
45  percent  of  the  teachers  in  the  longitudinal  sample  had  attained  this  level  of  education. 

Table  2.6 

Teacher  Characteristics— District-wide  vs.  RAND  Survey  Sample 
1997-1998  School  Year 


District  (n=329) 

Survey  Sample  (n=40) 

Male 

11% 

8% 

With  MA  degrees 

40% 

45% 

Average  years  teaching  experience 

13 

13 

White 

37% 

33% 

African  American 

15% 

20% 

Latino/Latina 

47% 

47% 

Asian  American 

0.3% 

None 

Native  American 

0.3% 

None 

Observations  and  Logs  of  Instructional  Activities 

In  the  spring  of  the  1997-1998  school  year,  RAND  conducted  classroom  observations  a 
subsample  of  12  teachers  from  the  larger  group  of  64.  These  observations  consisted  of  a  RAND 
researcher  shadowing  a  teacher  for  a  day,  writing  detailed  summaries  of  classroom  activities, 
taking  notes  on  the  general  description  of  the  classroom  and  the  resources  in  it,  and  informally 
discussing  design  team  activities  with  the  teacher. 

School  observations  first  began  in  the  spring  of  1998  and  continued  throughout  the  1998- 
1999  school  year.  Observations,  targeting  the  fourth  grade  level,  covered  10  different  schools. 


®  Each  teacher  who  participated  in  this  part  of  the  study  received  a  $200  honorarium. 
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Data  were  collected  in  two  Co-NECT,  two  BLOB,  two  MKSH,  two  SFA/RW,  and  two  non-NAS 
schools.  A  total  of  19  teachers  allowed  RAND  to  observe  their  classroom  activities  on  three  (in 
some  cases  four)  separate  occasions  and  interview  them  at  length  over  the  course  of  one  school 
year.  In  addition,  these  teachers  provided  work  assignments,  lesson  plans,  and  even  district 
memos  when  appropriate. 

We  also  aimed  to  gather  more  extensive  classroom  data  through  (1)  teacher  logs  of 
assignments,  homework,  projects,  quizzes/  tests/  exams,  and  papers  or  reports  over  a  five-week 
period,  and  (2)  illustrative  teacher-selected  samples  of  a  handful  of  students'  work  for  a  major 
project  during  the  spring  semester.  Because  we  could  not  gain  entree  into  these  classrooms 
until  May,  right  after  the  TAAS  testing,  and  because  our  logs  were  overly  burdensome,  the 
response  rate  for  these  12  teachers  providing  additional  information  via  logs  was  less  than 
desirable.  Five  of  12  teachers  [42  percent]  returned  completed  logs. 

Therefore  in  the  second  year,  we  significantly  revised  the  logs,  had  a  staff  person  on  site 
in  San  Antonio  to  encourage  teacher  participation  and  pick  up  the  logs  on  a  regular  basis,  and 
had  teachers  fill  out  logs  at  a  more  convenient  time  (i.e.,  early  fall). 

Interviews 

In  the  spring  of  1998,  we  also  conducted  focus  group  interviews  with  4*  grade  teachers 
from  eight  different  schools,  including  schools  implementing  each  of  the  four  NAS  designs  and 
some  comparison  schools.  These  interviews  help  us  better  understand  design  team  program 
characteristics,  instructional  strategies,  professional  development  activities,  and  classroom  level 
resources.  Further  information  about  these  schools,  professional  development  activities,  and 
the  resources  for  implementation  was  provided  by  forty-five  minute  structured  interviews  with 
principals. 

During  the  1998-1999  school  year  after  each  observation,  teachers  were  interviewed  about 
what  occurred  during  the  observation  as  well  as  other  more  general  issues  about  design 
implementation,  instructional  strategies,  professional  development,  and  other  matters  related 
to  design  and  district  programs  tiiat  were  affecting  teachers. 

In  addition,  we  conducted  interviews  of  NAS  design  team  leaders,  district  staff,  school 
instructional  leaders,  and  principals. 

STUDENT  DATA 

Data  for  students  were  obtained  mainly  through  district  cooperation  in  providing 
district  files  on  students  for  analysis. 
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student  Achievement 

In  this  study,  student  achievement  was  measured  in  a  variety  of  ways.  The  Stanford-9 
open-ended  reading  test  is  different  from  conventional  tests  because  it  allows  students  to 
explain  their  answers  with  open-ended  response  formats.  The  test  takes  about  50  minutes  to 
administer. 

In  addition,  RAND  obtained  the  Texas  Assessment  of  Academic  Skills  (TAAS) 
mathematics  and  reading  scores  for  all  3'^,  4*,  and  5*  grade  students  during  the  time  of  this 
study.  Our  focus  was  mainly  on  the  1997-1998  4*  grade  cohort  and  tracking  their  achievement 
back  to  when  they  were  3"*  graders  and  forward  through  the  5*  grade.  Specifically,  we  are 
analyzed  the  TAAS  mathematics  and  reading  Texas  Learning  Indices  (TLI).  These  data  were 
linked  to  teachers  and  schools  in  our  survey  sample.  They  allowed  us  to  examine  achievement 
across  schools  and  classrooms  for  the  entire  district  in  addition  to  the  RAND  sample  that 
included  teacher  surveys  and  Stanford-9  tests. 

Student  Characteristics 

Other  information  available  for  individual  students  from  district  data  files  included 
students  race-ethnicity,  sex,  date  of  birth,  poverty  status  (economically  disadvantaged  or  not), 
number  of  weeks  the  student  was  in  the  school  during  the  school  year,  limited  English 
proficiency  status,  and  participation  in  Special  Education  or  Talented  and  Gifted  programs. 

Examples  Student  Work 

The  teachers  we  observed  were  asked  to  provide  examples  of  students'  work.  We 
randomly  selected  a  quarter  of  the  students  in  each  class  every  three  months.  Once  a  student 
was  selected,  his  or  her  name  was  removed  from  the  class  roster.  While  no  criteria  were 
established  with  regard  to  what  was  submitted,  we  asked  teacher  to  provide  examples  of 
typical  work  assignments  students  produced. 

We  cannot  claim  that  the  submitted  work  was  representative  of  all  student  assignments 
made  by  a  given  teacher.  However,  diese  examples  did  provide  a  glimpse  of  the  types  of 
activities  assigned  by  each  of  the  teachers  in  our  sample. 

District  staff  assisted  RAND  in  selecting  teachers  academically  and  demographically 
representative  of  its  elementary  schools.  In  light  of  this  sample  selection,  our  findings  must  be 
interpreted  with  care.  The  small  number  of  schools  inspires  caution  as  does  the  even  smaller 
percentage  of  teachers  observed  and  interviewed.  Confidence  in  the  generalizations,  however, 
lies  in  the  fact  that  we  were  able  to  compare  some  our  results  with  the  teachers  and  students  in 
aU  district  schools.  Because  were  able  to  draw  on  a  variety  of  qualitative  and  quantitative  data. 
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we  were  able  to  compare  findings  from  these  data  sources  to  check  the  robustness  of  the 
findings  reported. 

In  addition  to  the  teacher  surveys,  test  scores,  and  other  quantitative  data,  our  multiple 
classroom  observations,  conversations  with  teachers  and  school  administrators,  examination  of 
lesson  plans,  and  analysis  of  student  work  revealed  that  design  implementation  is  greatly 
affected  by  the  environments  of  the  district  and  the  schools.  As  will  be  revealed  in  detail,  the 
designs  themselves  were  but  only  one  means  brought  in  by  the  district  to  reform  its 
academically-troubled  school  system. 
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3.  THE  DISTRICT  CONTEXT  FOR  IMPLEMENTATION  OF  NEW  AMERICAN  SCHOOLS' 

DESIGNS  IN  SAN  ANTONIO 


While  New  American  Schools  (NAS)  was  busy  starting  up  in  July  of  1991,  the  San 
Antonio  school  district  struggled  to  raise  its  students'  achievement  levels  and  meet  the 
challenges  it  faced.  Given  its  size  of  94  schools,  58,000  students,  and  3,800  teachers  at  the  time, 
productive  communication  proved  problematic  as  did  the  effective  utilization  of  district  staff. 
Much  energy  was  expended  on  the  management  of  day-to-day  organizational  affairs. 
According  to  several  central  office  administrators,  instructional  practice  was  too  often  last 
addressed.  In  the  words  of  one,  "The  school  district  was  perceived  as  backwater,  low- 
performing,  not  doing  anything,  in  decay." 

In  November  of  1993,  the  superintendent  of  San  Antonio  announced  his  retirement  plans, 
resulting  in  an  active  search  for  his  replacement.  The  search  committee  agreed  to  look  for  a 
leader  focused  on  instruction.  The  academic  situation  in  the  district  was  dire.  Forty-six  schools 
within  San  Antonio  were  deemed  "low-performing"  according  to  results  on  the  Texas 
Assessment  of  Academic  SkiUs  (TAAS),  a  criterion-referenced  test  covering  difference  subject 
areas  (particularly  reading  and  mathematics)  introduced  by  the  state  in  fall  of  1990.^ 

The  search  for  San  Antonio's  new  superintendent  was  narrowed  down  to  a  pool  of 
several  candidates.  The  winning  candidate  was  hired  in  a  close  vote  by  the  school  board  of  4  to 
3.  From  the  start,  the  new  superintendent  faced  tenuous  support. 

CURRICULUM  AND  INSTRUCTION 

Prior  to  the  new  superintendent' s  arrival  there  was  no  sense  of  a  imified  curricular 
vision  across  the  district,  let  alone  among  the  various  feeder  schools.  Individual  schools  had  in 
place  a  wide  variety  of  curricular  and  instructional  programs,  with  little  coherence  among 
them.  According  to  one  central  staff  member,  "We  had  a  curriculum  that  really  wasn't  going 
anywhere.  People  weren't  using  it."  When  school  staff  was  asked  what  instructional  strategies 
were  in  place,  a  typical  response  tended  to  include  twelve  to  fourteen  different  programs. 
Classrooms  basically  functioned  in  isolation.  Though  people  at  the  district  level  were 
responsible  for  the  various  programs,  there  was  no  expectation  for  entire  schools  or  even  a 
majority  of  classrooms  to  adopt  them. 

This  diversity  and  range  of  programs  across  school  campuses  made  it  difficult  to  know 
what  students  were  being  taught  and  how  learning  was  being  assessed  within  classrooms 


^See  http:  /  /wnvw.tea.state.tx.us/ studentassessment/. 
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across  the  district.  Moreover,  without  a  unified  curricular  trajectory,  the  same  topics  were  at 
times  observed  being  taught  at  a  variety  of  grade-levels. 

"We  had  a  lot  of  redimdancy  in  the  curriculum  and  we  had  a  total  lack  of  direction,  in 
part  because  each  school  in  this  district  very  much  did  its  own  thing. .  .1  walked 
(through)  a  third,  a  fifth,  a  seventh,  and  a  ninth  grade  classroom.  Within  the  same  ten- 
day  period  they  were  all  doing  the  solar  system....everybody  was  doing  exactly  the  same 
thing.  The  mobiles  were  hanging  in  every  room. .  .The  test  was  the  same." 

DISTRICT  CONTEXT  BEFORE  THE  NEW,  REFORM-MINDED  SUPERINTENDENT 

In  what  follows,  we  describe  further  the  district  context  before  the  new  superintendents 
arrival  in  terms  of  instructional  leadership,  professional  development,  and  parent  and 
community  involvement. 

Instructioital  Leadership 

The  focus  of  the  administration  prior  to  the  new  superintendent's  arrival  had  less  to  do 
with  issues  instructional  in  nature  than  managerial  and  administrational.  According  to  one 
district  administrator,  "In  previous  years,  the  important  thing  was,  have  we  got  all  the  kids  in 
place?  We're  taking  care  of  our  problems,  fighting  all  the  fires.  Are  we  spending  money  the 
right  way?  Those  were  the  kinds  of  things  we  checked  on." 

In  the  past,  the  organization  of  the  central  office  itself  in  fact  took  away  from  staffs' 
ability  to  focus  on  instructional  concerns  and  provide  the  necessary  assistance  to  its  schools. 


"All  the  elementary  principals,  65  of  them,  reported  to  one  person.  He  evaluated  every 
one  and  listened  to  die  parent  complaints  from  aU  65.  The  other  assistant 
superintendent  was  in  charge  of  all  the  middle  and  high  schools,  evaluated  every 
principal  and  listened  to  all  the  complaints.  And  then  they  had  attached  to  them, 
elementary  curriculum  and  instruction  and  secondary  curriculum  and  instruction, 
(respectively)... What  took  precedence  was  keeping  schools  rurming." 


The  assignment  of  responsibilities  within  the  central  office  was  not  designed  to  foster 
communication  among  staff,  thus  providing  little  opportunity  for  them  to  collaborate  with  one 
another.  As  stated  previously,  different  central  office  personnel  were  responsible  for  the 
various  programs  in  place  at  the  time  (e.g..  Cooperative  Learning,  New  Jersey  Writing  Style, 
Reading  Styles,  and  Visual  Math). 

In  some  cases,  it  was  difficult  for  central  office  staff  to  provide  anything  but  very 
specialized  help  to  schools  because  their  assignments  were  too  specific.  Moreover,  needed 
support  at  the  school  level  tended  to  vary  from  one  classroom  to  anofiier,  making  it  difficult  for 
any  one  staff  member  to  meet  the  multitude  of  demands  he  faced. 
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The  comments  made  by  several  other  central  staff  members  who  worked  in  the 
administration  prior  to  the  new  superintendent's  arrival  also  portrayed  die  image  of  a 
disjointed  system,  where  employees  worked  hard  and  did  their  best  in  isolation. 

"The  area  superintendents  were  doing  everything.  They  were  doing  curriculrun, 
instruction,  personnel  things  in  the  school,  I  mean,  it  was  crazy." 

"I  think  the  academic  support  team  members  were  tremendously  underutilized. 

Instead  of  working  on  curriculum,  they  each  had  a  specialty.  Like  one  did  cooperative 
learning,  one  did  learning  styles,  one  did  reading  styles.  They  each  had  these  little  areas 
of  specialization. .  .No  focus.  No  direction.  Just  piecemeal." 

"The  central  office  people  had  a  frustration  about  the  inability  to  meet  the  needs  of 
everybody  in  the  organization  in  their  narrow  field  that  they  were  working. . ." 

Within  schools,  given  that  teachers  engaged  in  a  variety  of  programs,  they  did  not 
necessarily  turn  to  one  another  for  assistance.  The  principals,  too,  were  of  little  help  in  the 
area  of  instruction.  Their  focus  tended  to  be  school  operations.  According  to  a  central  office 
administrator,  the  principals  pre-1994  were  "...brought  in  and  bred  as  managers.  The  message 
they  heard  was  manage  things  well.  They  were  not  instructional  leaders." 

Professional  Development 

In  addition,  the  professional  development  prior  to  the  new  superintendent' s  arrival  was 
limited,  according  to  central  staff  members  who  experienced  the  leadership  of  both  the  old  and 
new  superintendents.  The  district  did  its  best  to  meet  the  state's  yearly  staff  development 
requirements.  However,  it  had  no  good  system  of  tracking  professional  development. 
Moreover,  the  state  did  not  ask  for  specific  information.  Thus,  the  central  office  did  not 
develop  a  coherent  strategy  for  professional  development. 

Money  was  spent  on  upgrading  the  knowledge  and  skills  of  teachers,  but  "it  was  the 
one  shot  deal  with  no  real  follow-up  in  the  classroom  or  in  the  school,"  according  to  a  district 
staff  member.  Central  office  staff  reported  that  the  training  they  provided  tended  not  to  be 
utilized  at  the  school  level:  "There  was  a  whole  lot  of  training  that  was  happening  that  was 
wasted.  People  would  go  off,  learn  something  about  something,  come  back  to  their  schools, 
and  not  use  it."  Additionally,  administrators  at  the  central  office  recognized  that  although 
some  of  the  professional  development  they  provided  served  to  deepen  educators' 
imderstanding  of  specific  subjects,  they  were  not  related  in  any  way  or  tied  to  larger  issues  such 
as  state  standards  or  student  achievement.  In  the  words  of  a  district  administrator: 
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"In  1991,  what  the  central  office  was  about  doing  was  developing  teachers'  ability  to  use 
a  curriculum,  rather  than  actually  developing  the  curriculum  itself.  So  it  became  a  lot  of 
instructional  strategy  development  at  that  point  in  time.  And  the  more  we  did  that,  the 
more  people  demanded  it.  We  started  doing  some  things  that  were  about  cooperative 
learning  strategies  and  learning  styles  and  much  more  about  techniques  -  how  to  teach 
much  more  than  what  to  teach.  And  since  there  was  no  professional  development 
function  other  than  that,  that  became  really  more  of  a  professional  development 
function.  We  started  doing  lots  and  lots  of  training.  But  it  was  isolated.  It  was  pull 
out,  take  the  teacher  out  of  the  school,  give  them  some  training  in  this,  and  send  them 
back  to  the  school.  Actually  bring  them  back  again  and  do  some  follow-up  training. 

And  then  send  them  back  again.  But  we  didn't  have  the  capacity  to  do  any  kind  of 
follow-up  with  those  teachers  once  they  hit  the  school.  When  they  hit  the  school,  it  was 
gone." 

Parent  and  Conununify  Involvement 

As  for  school  interaction  with  parents  and  the  commvmity  at  large,  that,  too,  was 
problematic.  According  to  a  central  office  administrator,  "We  had  a  lousy  relationship  with  our 
parents.  It  was  one  of  those  "you  just  stay  outside  the  door.  It  was  really  adversarial.  And  our 
schools  were  not  welcoming. . .  We'd  done  some  tilings  aroimd  parent  involvement  and  parent 
education,  but  it  was  really  lip  service." 

DISTRICT  REORGANIZATION 

Upon  her  arrival,  the  new  superintendent  reportedly  talked  to  as  many  people  as  she 
could  in  San  Antonio  to  gain  a  deeper  imderstanding  of  the  schools'  needs.  Based  on 
conversations  with  principals  and  teachers,  the  experiences  of  central  office  staff,  and  the 
knowledge  she  brought  with  her,  the  superintendent  proceeded  to  draw  up  five  district  goals. 
The  goals  were  developed  to  support  and  realize  the  mission  statement  she  constructed  soon 
after  her  arrival: 


". .  .to  be  an  urban  school  district  where  all  students  are  achieving  above  state  and 
national  standards.  Where  they  exhibit  personal  growth  and  where  they  are  of  service 
to  others." 


The  district  aimed  to: 

•  Increase  student  achievement; 

•  Foster  collaboration  and  commimication; 

•  Strengthen  parent  and  community  involvement; 

•  Build  an  infrastructure  for  professional  development;  and 

•  Provide  appropriate  school  facilities  to  aU  students. 
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With  this  framework  in  place,  the  central  office  took  the  first  of  many  steps  to  move 
toward  its  goals.  To  ensure  that  the  district  could  effectively  support  its  campuses,  the 
superintendent  critically  examined  its  organization  as  well  as  the  leadership  in  place  at  schools. 

Instructional  Leadership  at  the  District  Level 

To  facilitate  the  realization  of  the  district' s  five  new  goals,  the  superintendent  set  out  to 
build  an  infrastructure  to  support  instruction.  Upon  learning  that  teachers  and  principals  could 
use  more  central  office  support  and  that  the  efforts  of  the  district  office  were  not  optimally 
coordinated,  the  new  superintendent  set  out  to  reorganize.  She  began  by  eliminating  certain 
central  office  positions,  creating  new  ones,  and  reallocating  resources  to  better  serve  schools. 
Her  vision  was  to  create  a  blend  of  site-based  and  central  operations  management. 

"You  know,  we  have  to  come  to  terms  with  what  really  makes  sense  to  be  consistent 

district-wide  and  what  really  the  schools  should  be  able  to  decide." 

She  felt  that  there  should  be  consistency  across  the  district  with  respect  to  operations 
such  as  uniform  policy,  transportation,  and  discipline.  The  superintendent  believed  all  schools 
should  focus  on  instructional  matters  as  much  as  possible. 

The  superintendent  reorganized  by  hiring  or  reassigning  staff  to  provide  schools  with 
instructional  leadership.  At  the  district  level,  four  people  were  hired  to  serve  as  Instructional 
Stewards,  or  area  superintendents.  Each  was  assigned  his  or  her  own  Learning  Community,  a 
specified  group  of  elementary,  middle,  and  high  schools.  The  primary  responsibility  of  the 
Instructional  Stewards  was  to  support  schools  and  provide  instructional  guidance.  They  were 
required  to  report  directly  to  the  superintendent,  who  held  each  accoimtable  for  the  learning  in 
his  or  her  respective  community.  One  Instructional  Steward  remarked  that  "Curriculum, 
instruction,  assessment,  is  what  we're  aU  about."  Another  stated  that  Instructional  Stewards 
were  "...responsible  for  supporting  the  principals,  of  evaluating  them,  of  helping  them  to 
determine  the  priority  needs  within  their  schools  and  supporting  them  in  accomplishing 
whatever  it  is  they  needed  to  accomplish." 

The  responsibilities  of  Instructional  Stewards  did  not  include  dealing  with  school 
operations  and  maintenance,  though  at  times  they  formd  themselves  engaged  in  such  work. 

"We  end  up  dealing  with  facilities  because  whatever  is  of  a  concern  to  the  principal 
really  becomes  a  concern  for  us." 
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In  addition,  each  steward  was  given  assignments  beyond  overseeing  schools.  For 
example.  Instructional  Stewards  were  required  to  sit  on  task  forces  (such  as  the  Bilingual  Task 
Force  or  the  Race  and  Class  Task  Force),  oversee  the  logistics  of  professional  development, 
and/ or  supervise  the  implementation  of  particular  NAS  designs. 

At  schools,  the  Instructional  Stewards  were  expected  to  provide  support  by  assisting  the 
analysis  of  school  data  such  as  TAAS  results  and  supervising  the  development  of  campus 
improvement  plans.  They  were  to  study  the  campus  plans  of  every  school  in  tiieir  respective 
Learning  Communities  to  assess  their  viability  as  well  as  commitment  to  San  Antonio  district 
goals.  Knowing  what  the  central  office  expected  to  see  and  aware  of  each  school's  special 
issues,  the  Instructional  Stewards  were  in  a  position  to  facilitate  commimication  and  to  assist 
schools  in  their  efforts  to  reach  instructional  goals.  From  the  start,  it  was  clear  that  much  of 
their  time  would  be  spent  in  schools. 

Instructional  Leadership  at  the  School  Level 

At  the  school  level,  in  order  to  reinforce  her  instructional  emphasis,  the  superintendent 
began  by  assessing  the  principals  in  place  and  replacing  those  who  tended  to  neglect 
instruction.  Because  the  school  board  chose  the  principals  prior  to  the  superintendent's  arrival, 
the  principal  appointments  tended  to  be  mired  in  politics.  The  superintendent  tried  to 
neutralize  this  by  establishing  screening  committees  at  the  school  level  that  included 
representatives  of  different  constituencies.  The  various  groups  made  their  recommendations, 
and  the  superintendent  looked  for  people  who  could  show  her  data,  interpret  it,  and  plan  a 
course  of  action  based  on  collected  information.  In  aU,  forty  new  principals  were  hired, 
replacing  many  who  left  of  their  own  volition. 

The  superintendent  then  created  Instructional  Guide  positions  to  further  facilitate 
communication  and  action  around  instructional  practice.  The  Instructional  Guides  tended  to  be 
master  teachers  deemed  highly  competent,  knowledgeable  of  curriculum  and  instruction,  and 
able  to  readily  communicate  and  anticipate  the  needs  of  others.  They  were  hired  to  be  their 
schools'  instructional  leaders,  managing  tasks  that  principals  didn't  have  time  for.  The 
Instructional  Guides  also  were  to  serve  as  liaisons  between  the  district  and  their  schools, 
communicating  the  central  office's  ideas  to  teachers  and  learning  the  various  district  initiatives 
to  take  back  to  tiieir  respective  campuses.  Instructional  Guides  were  to  facilitate  the 
implementation  of  all  new  programs  and  provide  teachers  with  in-house  follow-up  to 
professional  development.  According  to  one  central  office  administrator,  "It's  a  real  tight  rope, 
because  the  Instructional  Guides  must  be  the  confidante  of  their  principals,  and  they  have  to  be 
well-received  and  welcomed  by  the  teachers." 
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Instructional  Guides  were  given  much  credit  for  enabling  the  district  office  to  push 
forward  and  implement  ideas  very  rapidly.  Quarterly  meetings  attended  by  Instructional 
Guides  and  central  office  staff  served  to  further  the  budding  lines  of  communication.  During 
these  meetings,  the  guides  reportedly  discussed  what  was  working  at  their  schools,  what  upset 
teachers,  what  needed  to  be  improved  upon,  and  what  additional  support  systems  were 
necessary. 

Collaboration  and  Communication 

The  superintendent's  attempts  to  restructure  and  redefine  leadership  at  both  the  district 
and  school  levels  were  all  part  of  her  plan  to  build  tiie  infrastructure  necessary  to  support 
instruction  and  foster  collaboration  and  communication  within  the  district  Meetings  were 
arranged  between  the  central  office  and  principals  from  particular  Learning  Communities. 
Moreover  same  school-level  principals  regularly  met  together  as  well.  The  superintendent  held 
teacher  coffees  once  a  month  open  to  aU  who  wished  to  communicate  with  her  and  with  one 
another.  The  central  office  felt  it  very  important  that  teachers  have  multiple  opportunities  to 
get  together.  They  made  efforts  to  encourage  schools  to  create  time  for  their  teachers  to  sit 
down  together  and  discuss  issues.  The  central  office  also  created  teacher  networks  to  promote 
communication  across  school  lines.  Moreover,  plans  were  made  for  schools  to  be  connected  via 
the  World  Wide  Web. 

Collaboration  and  communication  were  further  facilitated  through  leadership 
committees  implemented  at  the  district  and  school  levels,  established  in  January  of  1994,  and 
referred  to  as  the  San  Antonio  Leadership  Team  (SALT)  and  Instructional  Leadership  Teams 
(ILT),  respectively.  Like  many  other  Texas  school  districts,  the  San  Antonio  school  district 
moved  to  give  its  schools  the  authority  to  decide  how  best  to  improve  their  respective 
educational  records.  The  premise  behind  the  district  leadership  team,  or  SALT,  was  to  increase 
the  representation  of  voices  involved  in  making  a  variety  of  school-related  decisions. 

According  to  the  district' s  1998-1999  Resource  Guide  for  School  Improvement,  "District 
and  campus  level  planning  and  decision-making  committees  were  established  to  provide  input 
and  assist  in  establishing  and  reviewing  the  district' s  and  campus's  educational  plans,  goals, 
performance  objectives,  and  major  classroom  instructional  programs.  Both  SALT  and  the  ILTs 
were  to  serve  exclusively  in  advisory  capacities  to  the  superintendent  and  principals, 
respectively,  addressing  such  areas  as  curriculum,  staffing,  budgeting,  school  organization,  and 
staff  development."  Each  committee's  members  consisted  of  a  team  of  selected  and  appointed 
teachers,  parents,  commimity  members,  and  business  representatives.  Additionally, 
paraprofessionals,  classified  staff  members,  and  students  sat  on  ILTs. 
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Curriculum  and  Instruction 

In  an  effort  to  raise  test  scores  and  ensure  that  all  students  learn  particular  skills  and 
increase  their  knowledge  base,  the  central  office  set  out  to  develop  an  instructional  framework 
to  guide  teachers.  Originally,  curriculum  and  instruction  were  the  responsibility  of 
Instructional  Stewards.  An  Office  of  Curriculum  and  Instruction,  overseen  by  the  associate 
superintendent,  was  created  in  1997  to  help  establish  a  coherent  academic  vision  for  the  district. 
The  central  office  wanted  to  make  sure  that  comparable  instruction,  in  line  with  state- 
developed  academic  standards,  could  be  foimd  occurring  at  the  same  grade  levels  across  all 
district  campuses.  In  die  words  of  one  central  office  administrator: 

"The  whole  issue  of  the  standards  came  about  after  our  reflection  that  that's  what  we 
needed.  We  decided  we'd  go  with  Texas,  essential  elements  and  skills.  But  we  also 
needed  something  as  a  district,  because  we  were  just  all  over  die  place." 

Moreover,  it  was  believed  that  a  district-endorsed  curriculum  could  help  alleviate  the 
difficulties  stemming  from  the  district' s  high  rate  of  student  mobility.  Additionally,  die  Office 
of  Curriculum  and  Instruction  could  work  to  ensure  that  all  teachers  (particularly  those  new  to 
the  field)  know  how  to  teach  core  subjects.  In  the  words  of  one  Instructional  Steward: 

"I  think  the  primary  reason  (the  Office  of)  Curriculum  and  Instruction  was  established 
was  we  re  a  district  of  teachers  who  have  been  here  25-30  years  or  very  young  teachers 
with  very  little  experience.  So  they  come,  especially  in  this  state  now,  with  the  inability 
even  to  teach  reading,  because  they  don't  know  how.  So  by  having  (an  Office  of) 
Curriculum  and  Instruction  focusing  on  that  and  teaching  these  teachers  how  to  teach, 
because  they  really  haven't  in  their  college  experience  had  that,  maybe  we  can  create  a 
critical  mass  of  teachers  with  experience  who  have  good  strategies  that  ultimately  with 
principals  and  community  could  get  the  kinds  of  learning  going  on  in  schools  that  these 
kids  need." 

The  Office  of  Curriculum  and  Instruction  soon  introduced  research-based  instructional 
programs  targeting  the  two  most  basic  subjects,  mathematics  and  reading.  For  mathematics,  the 
staff  decided  upon  the  program.  Everyday  MatJiemaHcsy^  Initially,  schools  voluntarily  took  on 
the  program.  In  the  spring  of  19%,  all  schools  were  told  by  the  central  office  to  adopt  it^i 

Everyday  Mathematics  (EDM)  is  a  K-12  curriculum  developed  by  the  University  of  Chicago  School 
Mathematics  Project  The  curriculum  assumes  a  spiraling  approach  to  instruction  where  students  are 
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A  reading  program.  Balanced  Literacy,  was  promoted  and  widely  implemented  during 
the  1998-1999  school  year.  The  program  provided  teachers  with  the  strategies  needed  to  teach 
reading  as  well  as  the  materials  necessary  to  enable  students  to  succeed.  Instructional  Guides 
were  made  responsible  for  ensuring  that  teachers  understood  the  program's  processes  and  used 
its  strategies. 

The  district  also  moved  to  incorporate  technology  into  instruction.  It  was  not  given  the 
same  attention  by  the  Office  of  Curriculum  and  Instruction,  however.  Because  the  district  office 
was  eager  to  see  its  schools  keep  pace  with  technological  advancements,  it  provided  teacher 
training.  Teachers  were  asked  to  teach  their  students  certain  computer  skills  and  utilize 
computers  as  much  as  possible.  The  district  developed  a  plan  to  provide  internet  access  to  all 
schools.  Classrooms  in  every  school  were  provided  with  several  computers. 

Professional  Development 

To  fully  support  instruction,  the  central  office  revamped  its  system  of  professional 
development.  Instructional  Guides  were  key  to  this  change.  As  stated,  they  were  seen  as  the 
primary  source  of  on-site  instructional  support.  In  the  words  of  one  district  administrator, 

"One  of  the  things  that  we  tried  to  do  in  supporting  professional  development  was  to 
make  sure  that  we  had  some  support  right  there  on  campus.  So  we  really  restructured 
Title  I  in  saying,  okay,  we're  going  to  put  a  person  on  every  campus  who's  major  job  is  to 
support  teachers  through  providing  professional  development,  doing  model  teaching, 
finding  the  materials  or  instructional  things  that  teachers  need." 

Those  involved  in  designing  the  district' s  professional  development  looked  to  research 
findings  to  define  "good"  professional  development.  Furthermore,  they  constructed 
professional  development  with  their  instructional  vision  always  in  focus:  "reforming  schools  to 


repeatedly  exposed  to  key  ideas,  in  greater  depth  each  time.  Key  features  of  the  Everyday  Mathematics 
curriculum,  described  in  teachers'  manuals,  include:  problem  solving  about  common-life  situations; 
sharing  ideas  tiirough  discussions;  daily  routines;  yearly  projects;  establishing  links  between  past 
experiences  and  explanations  of  new  concepts;  cooperative  learning  through  small  group  activities; 
practice  through  games;  ongoing  review  throughout  the  year;  informal  assessment;  and  home/ school 
partnership. 

After  being  in  place  for  three  years,  teachers  were  given  the  opportunity  to  vote  for  or  against  Everyday 
Mathematics,  In  the  spring  of  1999,  the  curriculum  was  voted  out  in  favor  of  a  more  traditional  math 
program. 
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become  places  where  things  are  connected  for  students."  They  also  made  sure  to  develop  a 
coherent  series  of  workshops  that  included  follow-up. 

Given  the  district's  renewed  focus  on  instruction  and  emphasis  on  instructional 
leadership,  the  central  office  required  principals  to  go  through  a  professional  development 
program  customized  to  meet  their  individual  needs.  According  to  one  district  office  staff 
member.  Instructional  Stewards  put  together  a  catalogue  from  which  principals  could  pick  and 
choose  training  sessions.  Additionally,  principals  were  required  to  attend  training  in  areas 
where  Instructional  Stewards  noticed  weaknesses.  At  times,  stewards  strongly  advised 
principals  to  receive  professional  development  in  certain  areas,  leaving  the  ultimate  decision  up 
to  them.  In  the  words  of  one  Instructional  Steward: 

". .  .we  ask  principals  where  they  want  to  grow.  We  interact  with  all  the  departments  and 
ask. .  .where  principal  training  is  needed  to  address  the  needs  of  the  department.  If  it's 
finance,. . .a  department  head  may  say,  half  of  them  turn  in  their  budgets  wrong.  Then 
we  know  they  need  training  in  the  finance  area.  Another  way  is  when  the  Instructional 
Stewards  work  with  principals  (on  a  specific  problem),  for  example  conflict  mediation. 
We  may  be  going  to  campuses  frequently  because  schools  are  faced  with  a  nxunber  of 
grievances  or  they  have  a  lot  of  parent  complaints.  Then  a  steward  would  say,  well  1 
need  for  my  principal  to  go  through  this." 

Similar  opportunities  were  developed  for  teachers.  Content  knowledge  became  a  focus 
for  the  central  office,  specifically  the  Office  of  Curriculum  and  Instruction.  Training  was 
developed  with  a  heavy  emphasis  on  mathematics  and  reading.  As  stated,  the  curriculum  and 
instruction  staff  felt  it  especially  important  to  provide  teachers  with  instructional  strategies. 
Furthermore,  they  were  determined  to  see  flie  district's  schools  meet  the  state's  standards  of 
instruction.  Technology  was  pushed  in  part  for  this  reason.  In  fact,  the  district  stipulated  in 
newer  teacher  contracts  that  teachers  become  skilled  in  the  use  of  technology  within  three  years 
time. 

Parent  and  Community  Involvement 

The  superintendent' s  administration  attempted  to  address  parent  and  commimity 
involvement  issues  as  well.  A  Parent  Community  Partnership  Network  was  formed  with  grant 
money  attached  to  the  Rockefeller  Initiative.  By  creating  an  office  to  support  parents,  the 
superintendent  aimed  to  further  enhance  the  infrastructure  needed  to  foster  high  quality 
instructional  delivery. 
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NEW  AMERICAN  SCHOOLS  IN  SAN  ANTONIO 

While  restructuring  instructional  leadership,  rethinking  fhe  delivery  and  content  of 
professional  development,  introducing  instructional  strategies  to  teachers,  pushing  state 
standards,  and  refocusing  the  district's  attention  on  instruction  and  student  achievement,  San 
Antonio  district  administrators  simultaneously  reviewed  national  reform  efforts  and  programs. 
One  idea  central  office  administrators  seriously  examined  and  eventually  decided  to  implement 
was  that  forwarded  by  New  American  Schools  (NAS),  namely  comprehensive  school  reform. 

Initially,  the  superintendent  introduced  one  NAS  design.  Expeditionary  Learning 
(BLOB).  This  occurred  soon  after  her  arrival.  It  was  a  design  she  had  overseen  while  serving  as 
superintendent  in  Dubuque,  Iowa.  In  fact,  she  had  been  influential  in  the  development  of  the 
ELOB  design  early  on  in  the  NAS  initiative. 

Soon  after  the  superintendent  came  to  San  Antonio  and  encouraged  the  district  schools  to 
implement  the  ELOB  design,  NAS  invited  San  Antonio  to  become  one  of  its  jurisdiction  sites. 
The  superintendent  committed  the  financial  resources  necessary  to  bring  the  NAS  designs  to 
the  district. 

The  superintendent  viewed  designs  as  the  needed  catalyst  to  force  schools  to  examine 
change  from  within.  She  did  not  want  the  piecemeal  practice  of  reform  to  continue  within  the 
district s  schools,  where  only  certain  classrooms  or  subject  teachers  engaged  in  new  practices. 
Not  only  did  she  view  the  NAS  designs  as  the  outside  galvanizing  force  for  change,  she  also 
had  hopes  that  the  designs  would  help  sustain  the  district's  efforts  to  engage  in  comprehensive 
school  reform.  Others  in  the  central  office  thought,  too,  that  the  NAS  designs  could  "..  .provide 
a  wholeness  and  integration  and  stimulate  teachers  to  think  or  rethink  what  they  were  doing." 
The  designs  also  were  seen  as  one  way  to  help  shift  teachers'  thinking  as  isolated  agents  of 
instruction  to  members  of  a  community  of  learners.  "When  you've  got  a  whole-school  design, 
everybody  plays,  everybody's  part  of  the  planning  process."  In  the  words  of  one  district  staff 
member: 

"You've  really  got  a  nice  framework  to  plan  for  all  the  professional  development,  and 
how  we're  going  to  deal  with  curriculum,  and  where  parents  fit  in,  and  what  kind  of 
support  system  we  need  to  have  and  what  are  the  materials.  It  just  really  is  such  a  help 
for  any  kind  of  school  to  really  finally  get  a  coherent  plan,  a  framework,  to  get  everybody 
moving  in  the  same  direction.  You  know,  we  got  everybody  working  hard  but  not 
always  moving  in  the  same  direction.  I  flunk  all  those  designs  are  a  wonderful  aid  to  us." 
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THE  INTRODUCTION  OF  NAS  DESIGNS  TO  SCHOOLS 

Convinced  that  the  designs  could  play  an  important  role  in  the  district's  efforts  to  bring 
about  increased  student  achievement,  the  central  office  approached  the  schools  via  the 
Instructional  Stewards.  Initially,  four  designs  were  selected  to  be  introduced  to  schools  within 
the  district  Modern  Red  Schoolhouse  (MRSH),  Expeditionary  Learning  (ELOB),  Co-NECT  (Co- 
NECT),  and  Success  for  All/Roots  and  Wings  (SFA/RW).  Orientations  in  the  form  of  site 
visits,  design  fairs,  and  design-centered  conversations  were  offered  to  familiarize  school  staff 
and  administrators  with  each  design's  philosophy  and  elements.  Design  literature  was 
distributed  as  well.  Schools  were  told  that  the  district  would  provide  the  financial  resources 
needed  to  implement  their  designs  of  choice.  During  the  first  roxmd  of  district-wide  design 
push,  the  schools  were  not  forced  to  select  a  design.  Teachers  were  told,  however,  that  within 
three  years,  all  of  the  district' s  schools  would  be  expected  to  have  one  in  place. 

Before  the  district  actually  became  a  NAS  jurisdiction,  the  central  office  required  that 
only  60  percent  of  a  school's  staff  be  in  support  of  the  only  design  introduced  at  that  time, 
ELOB.  During  the  second  phase  of  introduction,  the  office  raised  the  support  rate  to  70  percent. 
Still  the  district  found  that  those  who  opposed  adoption  were  great  enough  in  number  to 
significantly  impede  design  implementation.  Thus,  upon  becoming  a  NAS  jurisdiction  and 
introducing  schools  to  more  designs  at  the  end  of  the  1995-1996  school  year,  the  district 
required  diat  80  percent  of  any  school's  staff  be  in  favor  of  their  selected  design. 

Judging  by  the  accounts  of  teachers  and  other  school  staff,  decisions  to  adopt  particular 
designs,  as  well  as  the  reasons  for  doing  so,  tended  to  vary  by  school.  At  some  schools,  the 
principals  strongly  encouraged  their  teachers  to  consider  a  particular  design.  At  other  schools, 
teachers  were  directed  to  choose  what  they  thought  would  work  best.  Some  schools  felt  more 
pressure  than  others  to  decide  upon  a  design.  Many  teachers  expressed  that  they  had  been  told 
that  should  they  select  a  design  right  away,  their  school  would  be  guaranteed  the  money 
needed  for  implementation.  Many  teachers  said  that  they  felt  somewhat  rushed  to  make  a 
decision.  After  their  designs  of  choice  had  been  implemented  and  the  teachers  had  had  some 
time  to  reflect,  many  came  to  believe  that  they  did  not  have  sufficient  information  to  make  the 
decisions  they  did.  Some  expressed  that  they  were  not  aware  how  involved  and  labor- 
intensive  design  implementation  would  be.  Others  remarked  that  certain  aspects  of  their 
schools'  designs  appealed  to  them  more  than  others,  leading  them  to  make  the  decisions  they 
did.  In  some  cases,  this  worked  out  well.  In  others,  teachers  felt  that  their  imderstanding  of 
designs  had  been  faulty  from  the  start. 

The  adoption  of  SFA/RW  differed  significantly  from  the  other  designs.  Given  die 
district' s  focus  on  reading  (as  well  as  math),  the  reading  component  of  Roots  and  Wings  known 
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as  Success  for  All  (SFA)  drew  schools  to  this  design.  It's  not  clear  whether  all  components  of 
RW  were  even  introduced.  None  of  the  SFA/RW  schools  we  visited  implemented  more  than 
the  SFA  program.  Interviews  with  teachers  suggested  that  they  were  not  aware  if  plans  were  in 
the  works  to  implement  other  components  of  the  RW  design. 

THE  DISTRICT'S  ROLE  IN  SUPPORTING  COMPREHENSIVE  SCHOOL  REFORM 

Viewing  NAS  designs  as  the  framework  and  glue  to  tie  the  multiple  district  initiatives 
together,  the  central  office  expected  to  monitor  the  progress  of  design  implementation  and 
support  the  schools  in  their  efforts.  It  provided  instructional  guides,  professional  development, 
and  financial  resources.  For  schools  that  decided  upon  MRSH  and  CONNECT,  the  district 
expended  great  effort  to  wire  the  schools  quickly.  According  to  one  administrator,  "We  had 
always  put  dollars  in  to  try  to  cable  schools.  But  what  we  did  was  shift  them  to  make  sure  that 
our  priorities  were  always  project  Co-NECT  and  MRSH." 

As  time  passed,  it  became  clear  to  central  staff  members  and  design  team 
representatives  alike  that  greater  communication  was  needed  between  tiiem.  The  district  took 
the  initiative  by  arranging  quarterly  meetings  to  be  attended  by  all  design  representatives. 
Instructional  Stewards,  and  several  central  office  staff  members.  These  meetings  began  in  the 
1998-1999  academic  year. 

It  was  important  to  all  involved  to  determine  how  best  to  align  the  designs  with  the 
district's  plan  for  professional  development  and  emphasis  on  state-developed  academic 
standards.  There  had  been  confusion  regarding  this  because  in  some  cases  the  district 
initiatives  directly  conflicted  with  the  principles  of  various  designs.  Moreover,  when  tiiere  was 
overlap  between  district  and  design  ideas  regarding  instructional  practice,  the  teachers  often 
did  not  know  which  to  follow.  In  the  words  of  a  central  office  administrator: 

"I  understand  schools'  frustration  with  NAS  fitting  into  ti\e  district' s  plan  of  school 
improvement  because  it' s  hard.  It  is  hard  for  them.  And  I'm  not  sure. . .  it's  one  of  my 
biggest  stresses  right  now.  How  do  I  help  them?  I'm  not  sure  yet." 

It  was  important  to  the  central  office  that  schools  not  perceive  the  designs  as  "add-ons"  to 
current  district  initiatives. 

At  these  meetings  with  design  representatives,  the  central  office  strived  to  better  define 
its  role  in  the  design  implementation  process.  Discussions  revolved  around  design-generated 
benchmarks,  student  achievement,  and  curriculum,  for  example.  With  respect  to  benchmarks, 
the  central  office  wished  specifically  to  receive  design  team  assessments  regarding  their 
respective  views  of  implementation  progress.  However,  all  designs  except  SFA/RW  (already 
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an  exception  to  the  others,  given  tire  district' s  adoption  of  SFA  only)  were  reluctant  to  provide 
such  feedback  for  fear  of  jeopardizing  their  relationships  with  schools.  Eventually,  the  designs 
modified  their  positions  to  accommodate  the  district's  request. 

The  district  sought  assessments  of  implementation  progress  not  only  to  determine  the 
fruits  of  its  investment  in  NAS  designs,  but  also  to  determine  areas  of  weakness  in  schools' 
comprehensive  reform  efforts. 

IMPACT  OF  INCREASING  STATE  ACCOUNTABILITY 

Clearly,  the  operating  environment  into  which  NAS  designs  were  introduced  was  in 
great  flux.  The  district's  efforts  to  construct  an  infrastructure  for  professional  development  and 
to  foster  collaboration  and  commimication  among  district  administrators,  principals,  teachers, 
school  staff,  and  parents  rattled  the  status  quo,  causing  many  in  the  schools  to  react  tentatively 
as  they  adjusted  to  the  stresses  brought  about  by  change. 

The  emphasis  placed  by  the  district  on  increasing  student  achievement  and  aligning 
curriculum  with  state-endorsed  standards  furdier  escalated  the  pressure  that  teachers  felt  in 
their  efforts  to  make  sense  of  the  district  restructuring  and  to  learn  the  information  and 
practices  new  to  them.  Compoimding  this  pressure  was  the  increased  emphasis  on  their 
students'  TAAS  performance.  Given  the  low  performance  of  the  district's  students  on  TAAS, 
and  the  new  administration's  focus  on  increased  student  achievement,  the  push  to  increase 
standardized  test  scores  grew  ever  stronger  each  year. 

Pressure  was  attached  to  testing  success  given  the  use  of  TAAS  scores  to  determine 
school  ratings.  Ratings  of  "exemplary,"  "recognized,"  "academically  acceptable,"  and  "low 
performing"  were  made  based  on  TAAS  results,  annual  dropout  rates,  and  student  attendance. 
Student  performance  on  TAAS  was  not  only  compared  to  state  standards  but  to  schools' 
respective  performances  on  previous  exams.  The  following  table  defines  the  different  ratings. 


Table  3.1 
School  Ratings 


Exemplary 

Recognized 

Acceptable 

Low 

Minimum  Pass  Rate  (%)* 

90+ 

75+ 

35+ 

<35 

Dropout  Rate  (%) 

<1 

<3.5 

<6 

>6 

Attendance  Rate  (%) 

>94 

>94 

>94 

>94 

*Miniinuin  pass  rate  required  not  only  of  students  overall,  but  of  each  student  group  on  campus 
as  weU  (African  American,  Hispanic,  white,  and  economically  disadvantaged)  in  each  tested 
subject  area:  reading,  writing,  and  mathematics. 
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The  rating  of  individual  schools  annually  appeared  in  the  local  newspaper,  pushing 
school  staff  and  administrators  to  achieve  student  success  in  the  name  of  pride.  Moreover, 
"low  performing"  schools  faced  any  of  a  number  of  sanctions  listed  in  the  Texas  Education 
Code.  They  ranged  from  publicly  notifying  the  board  of  trustees  of  a  campus's  deficiencies  to 
appointing  a  board  of  managers  comprised  of  district  residents  to  exercise  the  powers  and 
duties  normally  assumed  by  the  district' s  board  of  trustees.  In  the  worst  cases,  where  schools 
had  been  found  low  performing  for  at  least  two  years,  campus  closure  was  threatened. 

The  district  obviously  wanted  to  see  its  schools  succeed  and  perform  well  on  TAAS. 
Moreover,  for  purposes  of  determining  its  own  accreditation  status,  the  district  desired  success 
in  its  own  right. 

A  financial  incentive  to  demonstrate  success  on  TAAS  was  introduced  by  the  state  in 
1995  imder  the  leadership  of  the  74th  legislature,  whereupon  the  75th  Texas  legislature 
appropriated  $5  million  for  dispersal  during  the  1997-1998  and  1998-1999  school  years.  The 
money  funded  the  Texas  Successful  Schools  Award  System  (TSSAS).^^  Based  on  the  three 
specified  criteria  explained  above,  TAAS  results,  low  dropout  rates,  and  high  attendance, 
schools  that  received  ratings  of  "acceptable,"  "recognized,"  or  "exemplary"  were  eligible  to 
receive  TSSAS  awards  ranging  from  $500  to  $5,000. 

SUMMARY 

The  multitude  of  changes  initiated  by  the  district  in  a  number  of  areas,  namely  school 
and  district  leadership,  professional  development,  commimication,  and  curriculum  and 
instruction,  propelled  teachers  and  school  administrators  to  critically  examine  organizational 
and  instructional  practices  and  to  engage  in  steps  needed  to  bring  about  envisioned  change. 
Simultaneously,  the  district  and  schools  felt  increasing  pressure  to  improve  TAAS  performance 
given  renewed  focus  on  instruction,  students'  low  scores,  impending  state  sanctions  for  poor 
performance,  and  desire  for  both  decent  campus  accountability  ratings  and  financial  rewards 
for  achievement. 

This  was  the  context  in  which  NAS  designs  were  introduced.  Given  the  many  reform 
initiatives  in  place,  as  well  as  the  speed  with  which  changes  were  occurring,  the  dynamic 
operating  environment  proved  over  time  to  be  less  supportive  than  necessary.  To  understand 
some  of  these  changes,  the  next  chapter  describes  further  the  school  context  m  which  NAS 
designs  were  implemented.  The  findings  are  based  on  our  analysis  of  our  longitudinal  teacher 
sample  (40  teachers),  observations,  interviews,  and  district  and  school  documents. 


^^See  http:/ /w^w.tea.state.tx.us/. 
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4.  IMPLEMENTATION  OF  NEW  AMERICAN  SCHOOLS  DESIGNS  WITHIN 
A  SYSTEM  OF  HIGH-STAKES  ACCOUNTABILITY 


In  the  spring  of  1998,  a  look  inside  San  Antonio  schools  and  classrooms  implementing 
New  American  Schools  designs  would  have  led  one  to  believe  that  the  transition  from  design 
selection  to  implementation  had  been  successful  to  date.  In  MRSH  schools,  for  example,  one 
saw  standards  posted  ubiquitously  on  classroom  walls  and  in  hallways  next  to  abtmdant 
displays  of  student  work.  In  SFA/RW  schools,  posters  of  hand  signals  for  classroom 
management  purposes  were  taped  to  the  hallway  walls.  Qassrooms  and  bulletin  boards  alike 
were  print-rich.  All  students  engaged  in  ninety  minutes  of  reading  instruction  at  the  same  time 
every  day  and  were  expected  to  read  for  twenty  minutes  every  night.  In  BLOB  schools,  posters 
of  the  design's  principles  were  found  in  classrooms  and  on  hallway  walls;  drafts  and  redrafts  of 
student  work  were  openly  displayed.  Teachers  developed  expeditions  into  which  they 
incorporated  project  work.  In  Co-NECT  schools,  students  utilized  computers  to  conduct 
research  and  frequently  worked  on  interdisciplinary  projects  in  small  groups. 

The  NAS  proposal  was  introduced  to  San  Antonio  schools  by  the  district  with  hopes 
that  the  marketed  "break-the-mold"  designs  would  provoke  teachers  and  administrators  to 

enthusiastically  in  comprehensive  school  reform.  The  thought  was  that  an  external 
model  provider  would  be  more  successful  at  pushing  and  sustaining  change  than  the  central 
office  could  ever  be  alone.  The  district  had  every  intention  of  fully  supporting  its  NAS  schools 
in  all  ways.  Not  only  did  it  plan  to  pay  for  the  services  of  design  teams  in  its  schools,  it  aimed 
to  provide  instructional  support  via  the  Instructional  Stewards  and  Guides.  Moreover,  the 
district  moved  towards  granting  its  schools  the  autonomy  to  make  school-specific  instructional 
decisions. 

Given  this,  in  1996,  when  NAS  designs  were  first  introduced,  one  might  have  expected 
that  in  time,  design  schools  would  look,  feel,  and  in  some  ways  function  differently  from  one 
another  as  well  as  from  schools  that  had  elected  not  to  take  on  comprehensive  school  reform 
models. 

However,  a  closer  examination  of  both  whole-school  and  individual  classroom  activities 
tended  to  reveal  a  more  complicated  story.  In  this  chapter,  we  focus  on  the  factors  in  NAS 
schools  that  were  important  for  design  implementation.  With  information  from  our  surveys 
(i.e.,  the  longitudinal  sample  of  40  teachers)  and  fieldwork,  we  focus  here  on  the  challenging 
educational  environments  tiiat  these  schools  faced,  the  high  stakes  accountability  system  in 
which  they  operated,  the  process  for  adopting  NAS  designs,  support  for  implementation 
including  training  and  professional  development,  principal  leadership  and  school  resources 
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devoted  specifically  to  NAS  design  implementation,  and  teacher  collaboration  and  support  of 
file  NAS  designs. 

NAS  DESIGNS  PROVIDED  ASSISTANCE  TO  CHALLENGING  SCHOOLS 

NAS  design  teams  assisted  more  challenging,  lower  performing  schools  in  San  Antonio, 
just  as  they  did  in  other  jurisdictions  (see  Berends,  1999;  Berends  and  Kirby  et  al.,  2001). 

In  Texas,  the  relative  performance  of  individual  schools  is  determined  each  spring  by  the 
Texas  Education  Agency.  Ratings  are  issued  based  on  a  combination  of  factors  that  include 
average  performance  on  the  TAAS,  attendance,  and  dropout  rates.  Schools  are  "low- 
performing,"  "acceptable,"  "recognized,"  or  "exemplary."  We  calculated  average  annual 
ratings  for  NAS  and  non-NAS  schools  from  the  1995  through  1999  as  shown  in  Figure  4.1;  for 
this  figure,  we  examined  all  schools  that  were  NAS  by  the  1998-1999  school  year  and  tracked 
their  ratings  back  in  time.  In  1995,  test  scores  throughout  the  district  placed  all  schools  below 
the  acceptable  ranking  (averages  for  NAS  and  non-NAS  schools  were  1.96  and  1.97, 
respectively).  The  state  officially  assigned  the  district  a  "warning"  status  tiiat  year,  putting  it 
on  notice  to  improve.  After  1995,  average  ratings  have  improved  at  a  higher  rate  for  non-NAS 
schools  than  for  NAS  schools.  By  1999,  the  rating  for  NAS  schools  was  2.10,  while  the  average 
rank  of  non-NAS  schools  was  2.24. 


Figure  4.1— Average  Texas  Education  Agency  Rankings  for  Schools  in  San  Antonio,  1995- 

1999 
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The  premise  that  schools  adopting  NAS  designs  tended  to  be  lower  performing  dran  the 
non-NAS  schools  was  borne  out  by  differences  in  teachers'  perceptions  of  their  students.  In 
both  the  1998  and  the  1999  surveys,  teachers  were  also  asked  to  rate  the  average  academic 
ability  of  students  in  their  classes  compared  with  "the  national  average."  In  1999,  seventy- 
seven  percent  of  the  NAS  teachers  reported  that  die  academic  ability  of  dieir  students  was 
below  the  national  norm  compared  with  36  percent  of  non-NAS  teachers  (see  Figure  4.2). 
Although  this  perception  of  both  NAS  and  non-NAS  teachers  increased  from  the  year  before, 
the  more  important  point  to  note  is  a  significantly  greater  percent  of  NAS  teachers  than  non- 
NAS  ones  reported  that  tiieir  students'  academic  ability  as  being  below  die  national  norm. 


Note:  Percentages  are  based  on  a  total  sample  size  of  40  teachers-26  NAS  and  14  Non-NAS. 


Figure  4.2 -Percentage  of  Teachers  Who  Reported  that  the  Academic  Ability  Level  of  Their 

Students  Was  Below  the  National  Norm 
in  NAS  and  Non-NAS  Schools,  Spring  1998  and  1999 


Teachers  also  reported  on  a  variety  of  factors  that  moderately  or  gready  hindered  their 
students'  academic  success,  including  lack  of  basic  skills,  student  motivation,  and  student 
discipline;  inadequate  preparation  in  subject  areas;  student  mobility,  inadequate  support  from 
parents,  and  student  motivation  (see  Table  4.1).  In  each  category  and  in  each  year, 
substantially  higher  percentages  of  teachers  in  the  NAS  schools  reported  that  specific  factors 
hindered  student  achievement,  indicating  the  types  of  educational  challenges  they  faced  in 
their  schools  and  classrooms. 
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Table  4.1 

Percentage  o£  Teachers  Who  Reported  that  the  Following  Factors  Moderately  or  Greatly 

Hindered  Student  Academic  Success 


1998 

1999 

NAS 

Non- 

NAS 

NAS 

Non- 

NAS 

Lack  of  basic  skills 

85% 

71% 

100% 

79% 

Inadequate  prior  student  preparation  in  the  subject  areas 

96% 

71% 

92% 

71% 

Lack  of  student  motivation 

81% 

57% 

81% 

57% 

Student  advancement  to  next  grade  without  meeting 
promotion  requirements 

85% 

64% 

73% 

64% 

Lack  of  student  discipline 

58% 

36% 

62% 

High  student  mobility  in  and  out  of  the  school 

85% 

43% 

73% 

Inadequate  support  from  parents  for  students 

73% 

43% 

77% 

57% 

Poor  student  attendance 

58% 

36% 

42% 

29% 

Note:  Percentages  are  based  on  teacher  reports  in  a  total  sample  size  of  40  teachers  — 

26  NAS  and  14  Non-NAS. 


Although  average  TAAS  test  scores  and  TEA  rankings  increased  for  both  NAS  and  non- 
NAS  schools  between  1998  and  1999,  teachers'  perceptions  of  their  students'  academic  abilities 
tended  to  become  more  negative.  In  general,  schools  that  chose  to  adopt  NAS  designs,  or  were 
encouraged  by  the  district  to  do  so,  tended  to  be  lower  performing  relative  to  other  schools  in 
the  district. 

PRESSURES  TO  IMPROVE  STATE  TEST  SCORES 

The  press  to  increase  student  achievement  and  improve  test  scores  in  San  Antonio 
schools  was  clearly  evident  during  the  time  of  our  study.  To  this  end,  the  district,  imder  the 
superintendent' s  leadership,  actively  engaged  itself  in  the  process  of  reorganizing  in  accord 
with  the  U.S.  Department  of  Education's  criteria  defining  comprehensive  school  reform.^^ 

Critical  components  of  schoolwide  designs  under  the  federal  Comprehensive  School  Reform 
Demonstration  (CSRD)  program  include  curriculum  and  instruction,  content  and  performance  standards, 
assessments,  professional  development,  school  organization  and  governance  issues,  and  parent 
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established  an  Office  of  Curriculum  and  Instruction  responsible  for  developing  a  sequential, 
standards  aligned  curriculum  across  grade  levels  in  all  schools  throughout  the  district.  The 
subjects  covered  on  the  Texas  Assessment  of  Academic  Skills  (TAAS),  namely  mathematics  and 
reading,  were  given  primary  attention.  Additionally,  the  district  partnered  with  New 
American  Schools  to  help  tighten  its  focus  and  to  encourage  school  improvement.  Suddenly, 
schools  were  not  only  exposed  to,  but  required  to,  implement  many  ideas  at  once,  naturally 
resulting  in  some  confusion  and  resistance  on  the  part  of  school  staff. 

The  emphasis  on  increased  student  achievement  not  only  called  for  greater  student 
learning,  it  heightened  the  district's  focus  on  improved  TAAS  performance  as  well.  Tied  to  the 
Texas  system  of  school  accountability,  TAAS  scores  provided  concrete  measures  of 
achievement  readily  reported  to  and  imderstood  by  administrators  and  teachers  alike.  The  act 
of  addressing  targeted  skills  enabled  educators  to  work  toward  specific  academic  goals  during 
a  time  of  great  change  in  the  district.  Successful  TAAS  performance  not  only  became  the  goal 
easiest  to  visualize  but  in  fact  the  goal  to  attain.  Schools  paid  a  price  for  this,  however. 


involvement.  According  to  the  program,  "a  comprehensive  school  reform  program  is  one  that  integrates, 
in  a  coherent  manner,  all  nine  of  the  following  components": 

1.  Employs  innovative  strategies  and  proven  methods  for  student  learning,  teaching,  and  school 
management  that  are  based  on  reliable  research  and  effective  practices,  and  have  been  replicated 
successfully  in  schools  with  diverse  characteristics; 

2. Has  a  comprehensive  design  for  effective  school  functioning,  including  instruction,  assessment, 
classroom  management,  professional  development,  parent  involvement,  and  school  memagement,  that 
aligns  the  school's  curriculmn,  technology,  professional  development  into  a  schoolwide  reform  plan 
designed  to  enable  all  students  to  meet  challenging  state  content  and  performance  standards  and 
addresses  needs  identified  through  a  school  needs  assessment; 

3. Provides  high-quality  and  continuous  teacher  and  staff  professional  development  and  training; 

4. Has  measurable  goals  for  student  performance  tied  to  the  state's  challenging  content  and  performance 
standards,  as  those  standards  are  implemented,  and  benchmarks  for  meeting  those  goals; 

5.  Is  supported  by  school  faculty,  administrators  and  staff; 

6.  Provides  for  the  meaningful  involvement  of  parents  and  the  local  community  in  planning  and 
implementing  school  improvement  activities; 

^.Utilizes  high-quality  external  technical  support  and  assistance  from  a  comprehensive  school  reform 
entity  (which  may  be  a  university)  with  experience  or  expertise  in  schoolwide  reform  and  improvement; 
8.1ncludes  a  plan  for  the  evaluation  of  the  implementation  of  school  reforms  and  the  student  results 
achieved;  and 

9.1dentifies  how  other  resources  (federal,  state,  local,  and  private)  available  to  the  school  will  be  utilized 
to  coordinate  services  to  support  and  sustain  the  school  reform. 

Schools  and  districts  have  considerable  latitude  in  developing  strategies  to  achieve  the  program's  goals, 
provided  these  strategies  encompass  the  nine  defined  components  listed  above.  The  legislation  provides 
a  list  of  school  reform  models  as  examples.  The  guidance  in  the  legislation  encourages  schools  to  consider 
other  models  (or  their  combination)  tiiat  have  evidence  of  effectiveness.  Indeed,  schools  can  create  their 
own  models  of  schoolwide  reform  by  using  a  combination  of  approaches  to  curriculum  instruction, 
assessment,  and  organization  reform.  The  aim  is  to  implement  effective,  improvement  strategies 

(see  http://wvvw.ed.gOv/office5/OESE/compreforml. 
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According  to  teachers,  the  focus  on  TAAS  tended  to  mute  creativity  and  channel  all  activities 
toward  preparation  as  the  test  approached.  Some  teachers  reported  preparing  their  students 
for  TAAS  from  day  one  of  the  school  year  by  incorporating  test-taking  strategies  and  TAAS 
vocabulary  into  their  lessons. 

"We  are  very  TAAS  focused  at  the  beginning  of  the  year.  A  lot  of  us  would  think  in  that 
direction  from  the  begiiming  when  you  start  learning  how  to  highlight  in  the  book  and 
pick  out  what  is  important.  There  are  a  lot  of  strategies  that  we  teach  that  start  off  from 
the  very  beginning  in  all  the  lessons."  (SFA/RW  school) 

"I  think  TAAS  takes  up  pretty  much  the  day,  and  I  think  as  teachers  we  get  bogged 
down  with  those  worksheets  and  don't  come  up  with  other  creative  ways  to  implement 
the  objectives  that  they  test  on  in  TAAS.  So  I  think  we're  very  worksheet-oriented 
because  I  think  when  tire  children  do  get  that  test  booklet,  it  won't  be  in  the  form  of  a 
game,  it  won't  be  in  the  form  of  a  project.  But  it  would  be  in  paper,  pencil  test."  (ELOB) 

"When  you  plan  and  divide  kids  up,  you're  always  getting  ready,  getting  prepared  for 
TAAS,  getting  mstructed  in  TAAS,  getting  the  kids  ready  for  it.  Your  instructional  time, 
you  know,  you're  heavy  into  the  reading."  (Co-NECT) 

In  addition,  lack  of  time  during  the  school  day —a  chronic  issue— became  even  more 
problematic  in  light  of  teachers'  needs  to  balance  TAAS  preparation  with  other  instruction. 
Many  teachers  reportedly  coped  with  the  multiple  demands  on  their  time  by  putting  aside 
other  activities  to  focus  almost  exclusively  on  TAAS  as  the  test  dates  grew  closer. 

"Come  January,  MRSH  is  over  here,  you  know,  on  the  side...From  January  through  the 
end  of  February,  which  is  when  we  have  our  writing  TAAS,  we  write  compositions...we 
write  all  day,  every  day  in  the  month  of  February.  So  then  MRSH  is  out  the 
window...Maybe  once  a  week  we  could  do  that,  but  you  can't  teach  a  rmit  once  a  week. 
And  so  it  just  doesn't  happen.  Okay,  as  soon  as  that's  over  (TAAS  writing),  like  March 
1®*,  then  we're  cramming  for  the  TAAS  formatted  math  and  reading... And  we  do  that  for 
two  months  solid." 

"Honestly,  (TAAS)  plays  a  very,  very,  very  big  roll  in  how  we  structure  time...Starting  in 
January,  that's  when  it's  pretty  much  drop  everything  but  the  reading  program,  drop 
everything  except  SFA  and  TAAS." 
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During  the  1998-1999  school  year,  schools  administered  as  many  as  four  district-directed 
TAAS  simulations,  after  which  teachers  were  required  to  analyze  the  results  and  pinpoint  their 
students'  weaknesses. 

"...you.. .have  assessments  schoolwide  that  you  have  to  do  and  figure  out  tiie  percentage 
of  students  passing  and  write  out  a  pass  plan  on  how  you  can  get  those  students  who  did 
not  pass  up  to  passing  mode.  You  have  to  turn  it  in,  a  sheet  with  every  student's  name 
as  to  what  objectives  they  have  passed  and  what  objectives  they've  failed." 

"We  give  a  TAAS  simulation  and  if  your  class  is  extremely  weak  in  a  certain  area,  it  is 
your  (the  teacher's)  responsibility  to  boost  that  one  target  area." 

In  many  classrooms,  bar  graphs  were  posted,  revealing  individual  students'  scores  on 
each  sub-test.  Interestingly,  two  low-performing  TAAS  schools  were  "encouraged"  by  the 
district  to  suspend  all  activity  that  did  not  directly  stress  TAAS  skills.  For  one  school,  this 
meant  neglecting  its  NAS  design  altogether.  Another  school  suspended  all  design  activity  after 
spring  break  to  prepare  intensively  for  TAAS.  According  to  teachers,  they  were  told  to  do  so  by 
the  school  administration,  who  received  this  "suggestion"  from  tihe  district. 

At  several  schools  Instructional  Guides  remarked  that  1998-1999  was  the  first  year 
teachers  were  explicitly  asked  to  "teach  to  TAAS."  The  administration  disliked  having  to  make 
such  a  request,  but  felt  that  their  schools  had  no  substantial  say  in  the  matter.  Schools  feared 
being  placed  on  lists  that  threatened  their  existence.  Moreover,  a  district  policy  enacted  at  the 
start  of  the  1998-1999  school  year  based  teachers'  evaluations  in  part  on  their  students'  TAAS 
scores.  In  the  words  of  an  Instructional  Guide: 

"Our  TAAS  scores  are  probably  not  where  they  should  be.  Therefore  some  other  people 
influenced  us  a  bit  to  do  different  things  versus  a  Co-NECT  model.  So  we  haven't  been 
able  to  focus  on  it  as  should  be...We  really  haven't  been  focusing  on  the  Co-NECT  model 
as  we  should  at  aIl...We  have  always  felt  very  strongly  about...teaching.  I  mean  reaUy  get 
in  drere  and  teach  reading  and  teach  math  and...not  spend  a  lot  of  time  on  how  to  take  a 
test  and  this  kind  of  thing.  And  we've  been  forced  to  turn  around  and  do  much  more  of 
that." 


According  to  a  teacher: 


-45- 


"(The  district)  has  just  about  threatened  to  disown  schools  who  were  doing  ELOB 
because  we  weren't  concentrating  on  TAAS.  And  TAAS  is  the  be-all,  end-all...But  we're 
seeing  scores  that  are  not  acceptable." 

Whether  or  not  schools  were  directly  told  to  focus  on  TAAS  preparation,  teachers  at  all 
schools  in  our  sample  reported  feeling  pressure  to  "teach  to  TAAS"  given  the  high-stakes 
nature  of  the  test. 

To  help  students  perform  better  on  TAAS,  teachers  not  only  spent  time  or  reviewing  the 
skills  that  would  be  tested,  but  also  the  art  of  test-taking.  This  included  teaching  test-taking 
strategies  and  exposing  students  to  test  vocabulary,  wording,  and  format. 

"And  then  we  practice  with  bubbles,  transferring  back  and  forth.  And  the)^ ve  got  to 
have  a,  b,  c,  d,  e,  and  f...  And  they  really  have  to  practice  and  practice  and  practice  with 
that.  And  I  don't  know  why  it  is  so  hard...I  would  like  to  know  why  it  is  so  difficult  for 
them  to  make  the  transfer."  (MRSH) 

"It's  how  to  read  and  xmderstand  what  it' s  asking  because  if  you  understand  what  the 
TAAS  is  looking  for,  you  can  figure  out  how  to  answer  it... As  we  teach  skills  we  teach 
strategies  with  it  and  figure  out  exactly  which  strategy  is  appropriate  for  this  question." 
(SFA/RW) 

"..we  try  to  format  anything  we  do  the  way  (TAAS)  is  given."  (SFA/RW) 

"...the  kids  are  not...familiar  enough  with  the  format  of  the  question  or  the  vocabulary." 
(Co-NECI) 

It  is  within  this  context  of  high-stakes  accountability,  challenged  schools,  and  high  expectations 
for  school  improvement  that  the  NAS  designs  were  introduced  to  and  implemented  in  schools. 

ADOPTION  OF  DESIGNS 

Upon  talking  with  teachers,  principals,  and  district  staff,  it  became  clear  that  the  process 
by  which  teachers  learned  about  NAS  designs  varied  somewhat  from  campus  to  campus. 
Teachers  at  some  schools  reported  being  exposed  to  all  the  designs  supported  by  the  district— 
Co-NECT,  ELOB,  MRSH,  and  SFA/RW.  Others  heard  about  only  a  select  few.  A  number  of 
schools  in  our  sample  sent  a  select  group  of  teachers  to  design  presentations.  These  teachers 
then  came  back  to  their  schools  to  share  what  they  learned  with  their  colleagues  so  that  all 


could  vote  on  their  design  of  choice.  Some  schools  had  teachers  visit  actual  design  schools  and 
report  back  to  their  colleagues.  In  some  cases,  teachers  listened  to  the  presentations  of  design 
representatives  at  their  own  schools.  In  the  MRSH  schools,  the  principals  introduced  the 
design  to  their  teachers  after  each  visited  a  demonstration  site.  Though  teachers  at  these 
schools  were  told  about  at  least  one  other  design,  MRSH  seemed  to  be  the  one  favored  as  it  was 
introduced. 

Regardless  of  the  number  of  designs  to  which  each  school  was  introduced,  all  teachers 
across  our  sample  were  given  the  opportunity  to  vote.  At  many  of  the  schools  exposed  to 
multiple  designs,  teachers  first  discussed  the  suitability  of  each  to  their  respective  campuses 
tiien  approved  the  design  most  favored  through  a  vote.  In  some  cases,  all  presented  designs 
were  listed  as  choices  to  be  voted  on.  Early  on,  in  accordance  with  the  district,  initially  at  least 
60  percent  of  all  teachers  and  school  staff  had  to  vote  in  favor  of  a  given  design  for  it  to  be 
implemented. 

Across  our  sample,  teachers  reported  feeling  pressure  to  choose  a  design.  Given  that  in 
time,  all  district  schools  would  have  to  take  on  a  design,  teachers  never  had  the  choice  to  reject 
design  adoption  altogether. 

I:  "Did  the  school  feel  pressure  to  adopt  a  design?" 

T:  "Yes.  Yes,  we  did.  We  were  pretty  much  told  either  pick  it  now  or  we'U  pick  it  for 
you  later."  (Co-NECT) 

T:  "Yes,  definitely.  We  were  pressured.  We  were  pressured  to  adopt  a  design."  (ELOB) 

T:  "Most  definitely.  Yes."  (MRSH) 

T;  "Right.  We  had  to  (choose  a  design)."  (SFA/RW) 

Not  only  was  there  pressure  to  take  on  a  design,  several  teachers  stated  that  they  were 
given  little  time  to  learn  about  and  decide  upon  a  comprehensive  reform  model.  According  to 
one  of  our  MRSH  teachers:  "I  remember  tiiat  it  was  a  rush,  rush  thing...and  I  know  that  at  the 
time  we  voted  on  it,  we  had  no  idea  what  it  was.. .All  we  were  told  was  the  teacher  would  have 
a  lot  of  input."  In  the  words  of  tire  Instructional  Guide  at  one  of  our  SFA/RW  schools,  "I  think 
there  could  have  been  more  time  exploring  the  designs,  but  on  tiie  other  hand  you  can  over¬ 
examine  maybe  some  things."  The  Instructional  Guide  at  one  of  our  ELOB  schools  stated, 
"Truthfully,  I  felt  that  we  could  have  and  should  have  looked  at  other  designs.  But  because  of 
the  time  constraints,  we  had  to  immediately  decide,  and  we  did  not  get  an  opportunity  to  look 
at  as  many  designs  as  there  are  out  there." 

Teachers  reported  choosing  designs  that  seemed  to  match  their  schools'  visions  and 
instructional  approaches.  For  many  this  meant  going  with  the  design  that  required  the  least 
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change.  Teachers  at  one  Co-NECT  school,  for  example,  stated  that  this  design  over  the  others 
suited  them  best  given  that  they  already  had  reworked  their  curriculum  and  were  imwilling  to 
rewrite  it. 

"As  a  staff  what  we  were  looking  for  was  something  that  would  fit  what  we  already 
have.. ..We  weren't  willing  to  chuck  all  the  work  that  we  had  already  done...Co-NECT 
allowed  us  to  keep  the  curriculum  that  we  had  and  perhaps  enhance  it  with 
technology." 

The  community  engagement  aspect  of  Co-NECT  appealed  to  teachers  as  well.  At  our 
other  Co-NECT  school,  teachers  already  employed  project-based  learning  and  technology  to 
some  extent,  making  this  design  more  appealing.  In  the  words  of  one  teacher, 

"...planning  projects  we've  always  done.  We've  always  worked  on  a  thematic  basis, 
even  before  Co-NECT  was  here  and  that  was  one  reason  why  we  chose  it  because  we 
felt  that  it  would  be  easier  for  us  to  implement." 

At  one  of  our  SFA/RW  schools,  teachers  stated  that  having  to  write  thematic-based  units 
turned  them  off  to  their  other  choices.  In  the  case  of  our  ELOB  schools,  the  design  principles 
and  project-based  approach  to  instruction  appealed  to  teachers  and  school  staff. 

Thus,  while  teachers  were  attracted  to  certain  aspects  of  NAS  designs  and  were  given 
the  opportunity  to  vote  to  adopt  a  particular  design,  the  time  constraints  to  make  a  decision 
inhibited  a  greater  understanding  of  what  teachers  could  accomplish  with  a  NAS  design  in 
place. 

DISTRICT  ASSISTANCE  FOR  DESIGN  IMPLEMENTATION 

All  schools,  regardless  of  being  NAS  or  non-NAS,  received  increased  support  for  teachers 
in  the  form  of  Instructional  Guides.  The  Guides  assumed  responsibility  for  handling  all 
curricular  issues  on  campus  and  for  keeping  abreast  of  the  latest  instructional  strategies  and 
techniques.  When  needed,  they  assisted  teachers  in  classrooms  by  modeling  skills,  for  example. 
Instructional  Guides  also  helped  to  identify  and  locate  resources.  Not  only  did  they  tutor  and 
test  students,  they  provided  training  to  school  staff  as  well.  Furthermore,  they  worked  closely 
with  their  respective  principals,  serving  to  facilitate  communication  between  teachers  and 
administrators.  Given  their  many  roles.  Instructional  Guides  tended  not  to  spend  as  much  time 
in  classrooms  as  they  would  have  Uked.  Many  reported  that  a  good  chunk  of  their  time  was 
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spent  away  at  training  sessions.  Instructional  Guides  at  NAS  schools  attended  both  district  in¬ 
services  and  design  training. 

When  asked  whether  the  district  supported  their  schools'  design  implementation  efforts, 
most  teachers  indicated  that  it  did  so  passively.  The  central  office  allowed  schools  to  choose 
from  a  selection  of  designs,  for  example.  Additionally,  it  did  not  dictate  how  to  proceed  with 
design  implementation.  Most  importantly,  the  district  provided  the  funds  to  enable 
comprehensive  school  reform.  Qear  to  teachers,  however,  was  that  the  central  office's 
emphasis  was  on  test  results.  Thus,  teachers  in  design  schools  were  required  to  implement  the 
district' s  mathematics  and  reading  initiatives  in  addition  to  their  reform  models  of  choice.  In 
this  way,  support  from  the  central  office  for  design  implementation  was  conditional. 

"It' s  left  up  to  the  campus  and  the  grade  levels  on  how...to  integrate  all  of  this 
information.  So  I  don't  want  to  say  that  the  district  doesn't  support  the  design.  They 
do,  but  they  support  just  as  much  the  things  that  the  district  is  implementing  onto  the 
campuses  as  well." 

PROFESSIONAL  DEVELOPMENT 

Professional  development  is  a  crucial  element  for  school  improvement  (Bodilly,  2001; 
Caret  et  al.,  1999).  One  of  the  challenges  facing  NAS  schools  has  been  that  districts,  not  schools, 
control  the  resources  for  professional  development.  Districts  also  differ  in  the  amount  of 
funding  available  they  have  to  focus  on  specific  professional  development  efforts  for  NAS 
design  implementation.  Moreover,  some  designs  stress  the  importance  of  specific  design  team 
training  for  implementing  the  designs  (e.g.,  MRSH  and  SFA/RW).  Others  (e.g.,  Co-NECT  and 
ELOB)  emphasize  the  importance  of  long-term  development  of  teachers'  capabilities  and 
professionalization,  which  in  turn  should  contribute  to  ongoing  school  improvement. 

Whatever  the  approach,  the  availability  of  resources  for  design  team  training,  district  training, 
and  overall  professional  development  efforts  for  design  implementation  remains  a  challenge 
within  districts  that  have  competing  goals,  objectives,  and  incentives  for  teacher  professional 
development. 

Design  Team  Assistance 

Besides  the  district  and  the  Instructional  Guides,  design  teams  provide  another  important 
source  of  support  for  implementation.  Design  teams  assist  implementation  by  providing 
schools  and  teachers  support  such  as  training,  professional  development,  and  materials 
(Bodilly,  1998;  Glennan,  1998).  Each  design  team  aims  to  provide  schools  and  teachers  with 
resources  to  assist  in  implementation,  especially  in  terms  of  commimication  between  design 
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team  members  and  school  staff,  and  design-related  professional  development.  Teachers' 
responses  to  our  surveys  provided  a  broad  picture  of  how  all  design  schools  were  progressing 
in  implementation,  and  changes  that  occurred  from  one  year  to  the  next.  For  instance,  in  1998 
a  relatively  high  proportion  of  teachers  in  the  NAS  schools  (58  percent)  agreed  that  their 
respective  design  teams  had  clearly  communicated  "its  program  to  school  staff  so  that  it  could 
be  well-implemented."  This  t5rpe  of  communication  improved  in  1999  (88  percent  of  teachers 
agreed  witiK  this  item). 

Training  by  Design  Teams 

As  for  the  actual  design  training,  however,  there  was  little  regular,  consistent  assistance 
provided,  according  to  teacher  interviews  across  design  schools.  Over  time,  there  was  even  less 
contact  between  teachers  and  their  respective  design  representatives. 

In  large  part,  this  had  to  do  with  the  fact  that  these  representatives  serviced  numerous 
schools,  making  it  difficult  for  them  to  be  attentive  to  any  one.  It  also  appears  that  from  the 
start,  strong  relationships  rarely  were  established,  making  it  unlikely  that  teachers  would  rely 
on  their  respective  design  representatives  for  external  technical  support  and  assistance.  In 
some  schools,  design  representatives  turned  over,  disrupting  what  rapport  had  been 
established.  Several  teachers  in  our  sample  saw  their  design  representatives  so  infrequently 
that  they  didn't  even  know  their  names.  The  SFA/RW  schools  should  have  received  the  most 
regular  design  assistance  given  that  each  had  one  facilitator  on  campus  to  meet  its  needs. 
Additionally,  tiie  program  included  a  series  of  implementation  visits  conducted  by  SFA 
corrsultants.  At  one  of  our  SFA/RW  schools,  however,  the  Instructional  Guide  took  on  the  SFA 
facilitator  role  as  well,  making  it  very  difficult  for  her  to  efficiently  address  issues  having  to  do 
with  the  design  team  reading  program. 

Few  design  representatives  entered  classrooms  on  a  regular  basis.  Teachers  reported  that 
visitors  to  classrooms  tended  to  be  district  staff.  The  teachers  were  given  little,  if  any,  outside 
"expert"  support  that  enabled  them  to  objectively  assess  dteir  progress  and  growth  as  design 
teachers.  Teachers  reported  that  when  in  need  of  help,  diey  tended  to  turn  to  their  colleagues 
or  Instructional  Guides  first.  Across  design  schools,  teachers  didn't  have  enough  interaction 
with  their  respective  design  representatives  to  feel  their  absence. 

In  addition,  teachers  reported  on  our  surveys  that  their  participation  in  design-related 
professional  development  meetings/  conferences  declined  from  one  year  to  the  next.  In  1998, 62 
percent  of  teachers  reported  participating  in  these  t3q)es  of  activities  more  than  twice  during  the 
past  12  montiis;  50  percent  of  teachers  reported  doing  so  in  1999.  The  percentage  of  teachers 
who  reported  attending  workshops  or  courses  related  to  the  NAS  design  also  decreased  from 
50  percent  in  1998  to  39  percent  in  1999. 
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In  part,  these  decreases  may  be  due  to  design  teams  emphasizing  teacher  training  more 
during  the  initial  stages  of  implementation.  However,  the  decline  may  also  be  a  signal  that  the 
level  of  implementation  itself  was  declining  in  these  schools  because  the  district  was  shifting  its 
focus  away  from  NAS  efforts. 

Consistent  with  the  survey  results,  our  interviews  and  observations  revealed  that 
teachers  at  ELOB,  Co-NECT,  and  MRSH  design  schools  saw  their  respective  design 
representatives  with  little  regularity —an  impediment  to  design  implementation.  Regardless  of 
their  schools'  adopted  designs,  teachers  reported  the  need  for  more  concrete,  hands-on  training 
that  would  enable  them  to  better  rmderstand  design  processes. 

A  teacher  at  a  Co-NECT  school  said  that  she  would  have  found  it  helpful  if  at  design 
training,  participants  were  given  concrete  ways  to  involve  the  community  in  their  school 
affairs,  actual  samples  of  Co-NECT  products,  and/ or  demonstrations  of  effective  instructional 
strategies  endorsed  by  Co-NECT,  for  example. 

Teachers  at  MRSH  schook  expressed  the  need  for  workshops  that  actually  demonstrated 
how  to  write  units  deemed  high-quality  by  MRSH  standards  and  by  the  district/  state 
standards.  They  felt  that  they  had  some  sense  as  to  what  to  incorporate.  However,  they  didn't 
always  know  how  to  judge  the  quality  of  their  work.  Moreover,  teachers  expressed  the  need 
for  workshops  that  actually  taught  Core  Knowledge,  a  critical  curricular  component  of  MRSH. 
Not  all  teachers  knew  much  about  Islam  and  the  Holy  Wars,  China,  or  the  Middle  Ages,  for 
example. 

Teachers  at  one  SFA/RW  school  remarked  that  they  would  have  found  it  helpful  to 
periodically  have  SFA/RW  representatives  sit  in  their  classrooms  and  provide 
feedback  afterwards.  This  way,  any  uncertainties  they  had  about  the  program  and  its 
implementation  could  be  immediately  and  effectively  addressed. 

Teachers  at  ELOB  schook  stated  that  they  would  have  liked  to  see  what  defines  a  high- 
quality  culminating  project  They  desired  more  assktance  with  respect  to  curriculum  writing 
as  well. 

District  Training  and  Professional  Development 

In  addition  to  training  by  design  teams,  teachers  at  NAS  schook  also  received  ffie 
dktrict's  professional  development,  as  did  their  colleagues  from  non-NAS  schook.  Much  of  the 
in-  service  professional  development  revolved  around  the  district's  reading  and  math 
initiatives.  Teachers  at  SFA/RW  schook  attended  reading  in-services  provided  by  the  design 
rather  than  the  dktrict.  More  workshops  having  to  do  wiffi  language  arts  were  offered  during 
the  1998-1999  school  year.  Teachers  attended  technology  training  and  workshops  concerning 
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state  standards  and  curriculum  alignment  as  well.  Relatively  speaking,  few  social  studies  or 
science  workshops  were  provided. 

Because  NAS  teachers  were  obhgated  to  attend  as  many  of  these  various  in-services  as 
their  colleagues  in  non-NAS  schools,  the  amount  of  training  activities  served  only  to  heighten 
frustrations.  All  of  the  designs  but  SFA/RW  required  teachers  to  develop  units  and  write 
curriculum.  While  encouraging  schools  to  implement  NAS  designs,  the  district  simultaneously 
constrained  their  ability  to  do  so  by  telling  teachers  what  to  teach  and  how. 

The  district  and  design  teams  did  not  tend  to  coordinate  their  efforts  with  respect  to 
professional  development,  so  teachers  were  left  on  their  own  to  merge  the  information  they 
received  from  each.  This  was  not  easily  done  without  modifying  the  essence  of  each  design. 
Not  only  did  this  effort  burden  teachers'  workload,  it  also  led  to  confusion  as  to  what  to 
prioritize. 

On  our  surveys,  we  asked  several  questions  that  addressed  teacher  participation  in 
professional  development  activities  not  specifically  related  to  the  NAS  designs.  In  general, 
compared  with  NAS  teachers,  a  higher  proportion  of  non-NAS  teachers  reported  participation 
in  a  wide  variety  of  professional  development  activities  covering  topics  such  as  alternative 
student  assessments,  content  standards,  single  subject  teaching  areas,  teaching  techniques, 
classroom  management,  and  educational  technology.  Respondents  from  the  NAS  schools 
indicated  greater  levels  of  participation  in  fewer  activities:  student  portfolio  development, 
assessment,  and  site-based  management. 

One  survey  item  asked  teachers  the  extent  to  which  students'  academic  success  was 
hindered  by  a  lack  of  coherent,  sustained  professional  development.  As  indicated  in  Figure  4.3, 
31  percent  of  the  NAS  teachers  reported  that  a  lack  of  coherent  professional  development  was  a 
moderate  or  great  problem  in  hindering  students'  academic  success;  this  declined  somewhat  to 
27  percent  in  1999.  Non-NAS  teachers  reported  similarly  on  this  measure  in  both  1998  and 
1999. 
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Note:  Percentages  are  based  on  a  total  sample  size  of  40  teachers-26  NAS  and  14  Non-NAS. 

Figure  4.3—  Percentage  of  Teachers  Who  Reported  that  a  Lack  of  Coherent,  Sustained 
Professional  Development  Hindered  Students  Academic  Success  Moderately  or  Greatly  in 
NAS  and  Non-NAS  Schools,  Spring  1998  and  1999 

Several  survey  items  asked  teachers  about  the  quality  of  the  professional  development 
they  received  in  tiie  past  year.  Teachers  indicated  the  extent  to  which  they  agreed  or  disagreed 
(4  point  scale)  that  overall,  "the  professional  development  activities  sponsored  by  the  NAS 
design  team,  your  school  or  the  district,"  during  the  past  12  months  had: 

•  Been  sustained  and  coherently  focused,  rather  than  short-lived  and  unrelated; 

•  Included  enough  time  to  think  carefully  about,  try,  and  evaluate  new  ideas; 

•  Been  closely  connected  to  the  design  team's  activities  in  their  school  (NAS 
respondents  only); 

•  Included  opportunities  to  work  productively  with  colleagues  in  their  school; 

•  Deepened  their  understanding  of  the  subject  matter  they  teach; 

•  Led  them  to  make  changes  in  their  teaching; 

•  Helped  their  school's  staff  work  together  better; 

•  Changed  the  way  teachers  talk  about  students  in  their  school; 

•  Deepened  their  understanding  of  how  students  learn  the  subject  matter  they  teach; 

•  Altered  approaches  to  teaching  in  their  school. 

We  combined  these  items  into  a  composite  measuring  the  overall  quality  of  professional 
development.!^ 

The  alpha  reliability  of  this  quality  of  professional  development  composite  was  .92  for  1998  and  .94  for 
1999.  The  range  of  correlations  for  flie  individual  items  was  .34  to  .38  in  1998  and  .45  to  .50  in  1999. 
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Overall,  less  than  40  percent  of  all  surveyed  teachers  reported  that  they  agreed  or 
strongly  agreed  that  the  professional  development  they  received  was  of  high  quality  (see  Figure 
4.4).  In  1998,  38  percent  of  the  NAS  teachers  reported  that  they  agreed  or  strongly  agreed 
across  the  items  in  the  professional  development  composite  compared  with  27  percent  of  non- 
NAS  teachers.  In  1999,  NAS  teacher  reports  declined  slightly  (36  percent),  while  the  non-NAS 
teacher  reports  increased  slightly  (29  percent). 


Note:  Percentages  are  based  on  a  total  sample  size  of  40  teachers— 26  NAS  and  14  Non-NAS. 

Figure  4.4— Percentage  of  Teachers  Who  Agreed  or  Strongly  Agreed  with 
High-Qiiality  Professional  Development  Composite  Items 
in  NAS  and  Non-NAS  Schools,  Spring  1998  and  1999 


PRINCIPAL  LEADERSHIP 

When  considering  implementation  of  NAS  designs,  an  important  school  resource  to  note 
is  the  leadership  of  the  principal.  Other  studies  have  shown  the  importance  of  leadership  in 
establishing  effective  school  improvement  efforts  (Edmonds,  1979;  Brookover  et  al.,  1979; 

Pur  key  and  Smith,  1983;  Berends  and  Kirby  et  al.,  2001).  Based  on  our  surveys,  we  found 
differences  between  NAS  and  non-NAS  teachers  perceptions  of  principal  leadership  in  dreir 
schools.  For  the  principal  leadership  composite,  teachers  were  asked  the  extent  to  which  they 
agreed  or  disagreed  (on  a  4-point  scale)  with  die  following  statements;!^ 

•  The  principal  lets  staff  members  know  what  is  expected  of  them; 

The  alpha  reliability  for  the  principal  leadership  composite  was  .90  for  1998  and  .84  for  1999.  The  range 
of  correlations  for  the  individual  items  was  .38  to  .47  in  1998  and  .26  to  .30  in  1999.  More  positive  scores 
on  this  composite  reflect  more  positive  perceptions  of  teachers  about  principal  leadership  in  their  school. 
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•  The  school  administration's  behavior  toward  the  staff  is  supportive  and  encouraging; 

•  The  principal  does  a  poor  job  of  getting  resources  for  this  school  (reverse-scored); 

•  My  principal  enforces  school  rules  for  student  conduct  and  backs  me  up  when  I  need  it; 

•  The  principal  talks  with  me  frequently  about  my  instructional  practices; 

•  In  this  school,  staff  members  are  recognized  for  a  job  well  done; 

•  The  principal  has  confidence  in  the  expertise  of  the  teachers. 

Less  than  40  percent  of  both  the  NAS  and  non-NAS  teachers  in  botii  years  agreed  or 
strongly  agreed  that  their  principals  were  strong  leaders  according  to  these  items  (Figure  4.5). 
However,  in  both  years  teachers  in  NAS  schools  reported  higher  scores  on  this  principal 
leadership  composite  than  teachers  in  non-NAS  schools  by  20  percentage  points  in  1998  and  15 
percentage  points  in  1999. 


■  NAS 
□  Non-NAS 


Note:  Percentages  are  based  on  a  total  sample  size  of  40  teachers— 26  NAS  and  14  Non-NAS. 


Figure  4.5— Percentage  of  Teachers  Who  Reported  that  They  Agreed  or  Strongly  Agreed  with 

Principal  Leadership  Composite  Items 
in  NAS  and  Non-NAS  Schools,  Spring  1998  and  1999 


SCHOOL  RESOURCES  SPECIHC  TO  NAS  IMPLEMENTATION 

In  addition  to  principal  leadership,  another  important  factor  to  consider  regarding  the 
implementation  of  NAS  designs  is  whether  the  school  has  the  resources  available  to  do  so 
(Berends  and  Kirby  et  al.,  2001;  Berends,  2000).  NAS  teachers  were  asked  to  indicate  the  extent 
to  which  their  schools  had  the  resources  needed  to  implement  the  major  elements  of  their 
respective  designs  using  a  5-point  scale  ranging  from  "no  resources  are  available"  to  "all  are 
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available."  A  resources  composite  was  constructed  using  a  summative  scale  of  responses  to 
the  following  items:^^ 

•  Materials  to  support  instruction; 

•  Availability  of  design  team  materials  to  further  implement  tire  design  in  your  school; 

•  Professional  development  for  teachers; 

•  Time  for  planning,  collaboration,  and  development; 

•  Staff  or  consultants  to  mentor,  advise,  and  provide  ongoing  support; 

•  Technology  and  connectivity;  and 

•  Funds  and  funding  flexibility. 

According  to  teachers,  the  resources  available  to  them  improved  over  time:  42  percent  agreed 
tiiat  most  or  all  were  available  in  1998,  and  88  percent  agreed  in  1999. 

TEACHER  COLLABORATION 

In  addition  to  these  school  resources,  the  implementation  process  for  NAS  designs 
hinges  on  teachers  working  together  (Bodilly,  2001).  The  extent  to  which  teacher  coordinate 
their  efforts,  learn  from  each  other,  and  work  together  to  develop  and/ or  teach  the  curriculum 
is  critical  for  whole-school  change  (Louis,  Kruse,  and  Marks,  1996;  Marks  and  Louis,  1997; 
Newmann  &  Associates,  1996). 

Because  schools  adopting  NAS  designs  were  expected  to  engage  in  teamwork,  resulting 
in  increased  levels  of  cooperation  among  school  staff,  we  measured  teacher  collaboration  with  a 
composite,  combining  several  survey  items.  This  composite  was  constructed  from  items  that 
included  a  variety  of  response  formats  in  which  teachers  were  asked  the  extent  to  which  the 
program  elements  described  their  schools  (6-point  scale  ranging  from  does  not  describe  my 
school  to  clearly  describes  my  school),  the  extent  to  which  items  had  changed  in  the  past  three 
years  (5-point  scale  ranging  from  greatly  improved  to  greatly  worsened),  and  the  extent  to 
which  they  agreed  or  disagreed  with  statements  (4-point  scale  ranging  from  strongly  disagree 
to  strongly  agree).  The  following  statements  comprise  the  standardized  teacher  collaboration 
composite;^^ 

•  Teachers  are  continual  learners  and  team  members  through  professional 
development,  common  planning,  and  collaboration; 

•  Teachers  learning  from  one  another  (worsened  or  improved); 

f  ^  The  alpha  reliability  for  the  resources  composite  was  .87  for  1998  and  .91  for  1999.  The  range  of 
correlations  for  the  individual  items  was  .49  to  .63  in  1998  and  .58  to  .66  in  1999. 
f  ^  The  alpha  reliability  for  the  teacher  collaboration  composite  was  .84  for  1998  and  .85  for  1999.  The 
range  of  correlations  for  the  individual  items  was  .39  to  .53  in  1998  and  .42  to  .49  in  1999.  Some  items 
were  reversed  scored  so  that  a  positive  value  on  this  composite  reflected  greater  teacher  collaboration. 
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•  Most  of  my  colleagues  share  my  beliefs  and  values  about  what  the  central  mission  of 
the  school  should  be; 

•  There  is  a  great  deal  of  cooperative  effort  among  the  staff  members; 

•  Teachers  respect  other  teachers  who  take  the  lead  in  school  improvement  efforts;  and 

•  Teachers  at  this  school  respect  those  colleagues  who  are  experts  at  their  craft. 

Overall,  there  was  not  much  difference  in  the  degree  of  teacher  collaboration  among 

teachers  in  NAS  schools  compared  with  those  in  non-NAS  schools.  Since  the  items  in  the 
teacher  collaboration  composite  were  not  based  on  a  common  response  metric,  we  report 
whether  teacher  responses  for  this  composite  were  positive  or  negative  on  the  standardized 
scale  combining  the  different  survey  items.  In  both  1998  and  1999, 54  percent  of  NAS  teachers 
scored  positively  on  this  composite.  Exactly  50  percent  of  the  non-NAS  teachers'  scores  were 
positive  in  1998.  This  rose  to  57  percent  in  1999. 

Our  interviews  revealed  that  teachers  generally  praised  the  designs  for  nurturing 
collaboration  among  school  staff.  The  designs  themselves  were  imable  to  dissolve  the  teacher 
factions  that  existed  at  several  schools,  but  they  did  bring  about  more  teacher  interaction  and 
sharing  of  ideas  according  to  our  interviews  with  teachers.  Additionally,  the  implementation 
of  designs  enabled  teachers  to  be  more  aware  of  what  their  colleagues  at  other  grade  levels 
were  responsible  for  teaching.  Consequently,  there  was  less  overlap  of  subject  material  from 
one  grade  level  to  the  next.  A  greater  sense  of  unity  developed  among  teachers  as  well  with 
respect  to  curriculum,  instructional  strategies,  and  educational  mission. 

"...I  find  that  that's  one  thing  it's  done  is  that  we  get  together  a  whole  lot  more.  Before 
you  were  pretty  much  in  your  room  and  you  accomplished  it  and  tire/d  come  out  and 
say  here's  your  thing,  and  you  know,  you  did  it.  But  you  didn't  know  what  the  others 
had  done.  You  know,  but  now  we're  always...meeting  and  discussing  and  sharing 
papers...So  that  to  me  has  helped  a  lot  because  you  don't  feel  like  you're  out  there  all 
alone."  (Co-NECT) 

"I  think  on  a  grade  level  maybe,  it  gives  us  time  to  work  together.  Something  to  work 
for."  (ELOB) 

"...getting  the  teachers  better  acquainted  and  mixed  together  so  that  we  don't  have 
factions  in  the  building,  like  we  used  to.. .And  that  (the  MRSH  design)  has  been  kind  of 
a  common  ground  that  we've  all  met  on  and  that's  helped."  (MRSH) 
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"...it  (the  design)  is  promoting  more  getting  together  within  the  grade  level,  and  it  opens 
up  all  of  the  lines  of  communication  there,  but  I  think  grade  level  by  grade  level  it 
changes..."  (MRSH) 

It  is  difficult  to  know,  however,  whether  the  designs  themselves  were  solely  responsible 
for  such  changes.  Soon  after  her  arrival,  the  superintendent  aimed  to  build  collaboration  and 
communication  not  only  between  the  district  office  and  schools,  but  also  within  schools 
tiiemselves.  She  encouraged  every  school  to  practice  site-based  management,  pushing  the 
institution  of  Instructional  Leadership  Teams  (ILTs),  advisory  groups  designed  to  address 
curricular,  staffing,  budget,  organizational,  and  professional  development  issues.  In  essence, 
the  teams,  comprised  of  teachers,  parents,  community  members,  business  representatives, 
paraprofessionals,  classified  staff  members,  and  students,  were  responsible  for  overseeing  the 
production  and  implementation  of  dieir  respective  schools'  campus  improvement  plans.^^ 

In  reality,  NAS  schools  were  not  all  that  different  from  non-NAS  schools  in  that  common 
planning  period  tended  to  be  used  similarly.  At  all  schools  in  the  district,  teachers  were  given 
one  weekly  grade-level  planning  period  to  promote  collaboration. 

At  NAS  schools,  schedules  were  arranged  to  allow  for  double  planning  periods. 

Teachers  were  to  use  this  time  to  write  curriculum,  discuss  projects,  gather  resources,  and 
develop  units.  NAS  teachers  complained  that  though  they  were  given  more  time  to  work  with 
one  another,  in  actuality,  they  were  often  impeded  from  doing  so  by  having  to  discuss  a  variety 
of  other  issues  typically  district-generated.  Planning  time,  too,  was  frequently  let  go  to  address 
district  issues.  NAS  teachers  also  reported  not  meeting  at  times  to  take  care  of  such 
administrative  business  as  paperwork,  bookkeeping,  and  testing.  As  one  teacher  stated; 

"The  paperwork  has  multiplied  like  ten-fold  it  seems  like.  Every  time  we  turn  around 
we  have  to  do  a  paper...I  mean  we  have  to  write  everything  the  kids  are  doing  and 
where  they  are  low,  where  they  are  not  low,  what  they  need,  what  they' ve  done,  how 
you  addressed  it,  what  still  needs  to  be  done...There's  always  something  coming  from 
the...district  office  that  we  have  to  fill  out....Everyone  is  feeling  overwhelmed." 

According  to  the  Texas  Education  Agency,  campus  planning  serves  to  decentralize  the  decision¬ 
making  process.  Schools  determine  how  to  improve  student  performance  through  collaboration. 
Togetiier,  principals,  teachers,  campus  staff,  parents,  and  community  and  business  representatives  assess 
tire  educational  outcomes  of  students,  determine  performance  objectives  and  strategies,  and  ensure  that 
strategies  are  implemented  and  adjusted  to  improve  student  achievement  for  all  students.  The  Campus 
Improvement  Planning  Document  includes  four  main  sections:  (1)  the  introduction;  (2)  needs  assessment 
component;  (3)  goals,  objectives,  and  strategies;  and  (4)  activities  and  evaluations. 
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TEACHER  SUPPORT  FOR  THE  NAS  DESIGNS 

Over  the  two  school  years  we  conducted  our  research,  one  indication  of  changes  in  NAS 
implementation  came  from  an  item  that  asked  teachers  how  strongly  they  supported  or 
opposed  the  NAS  design  team  program  in  their  school.  In  1998, 54  percent  of  teachers 
indicated  that  they  "strongly  support  a  NAS  design  team  program"  in  their  schools,  but  this  fell 
to  25  percent  in  1999,  The  proportion  of  teachers  indicating  that  they  strongly  opposed  or 
somewhat  opposed  NAS  designs  in  their  school  increased  from  15  percent  in  1998  to  43  percent 
in  1999. 

Clearly,  the  central  office  played  as  active  a  role  in  initiating  change  across  the  district  as 
did  design  teams  in  their  select  schools.  The  actions  of  the  central  office  made  it  difficult  for 
NAS  teachers  to  view  design  implementation  as  a  district  priority.  Consequently,  these 
teachers  were  not  able  to  fully  commit  to  the  ideas  described  in  their  respective  design 
literature.  Some  feared  that  the  NAS  initiative,  too,  like  many  others  that  had  been  introduced 
over  the  years,  would  fade  away  in  time.  Furthermore,  aspects  of  designs  such  as  ELOB, 

MRSH,  and  Co-NECT  overwhelmed  many  teachers.  The  task  of  writing  curriculum  was  not  an 
activity  readily  undertaken  or  easily  accomplished  by  many,  given  their  lack  of  time  and 
experience. 

During  our  interviews,  teachers  reported  variation  with  respect  to  levels  of  design 
implementation  within  their  schools.  Implementation  in  individual  classrooms  depended  in 
large  part  on  teachers'  feelings  for  designs,  their  willingness  to  invest  time  and  energy,  and 
their  particular  strengths  and  weaknesses.  One  teacher  in  our  sample  stated  that  within  her 
school,  differing  levels  of  competency  existed  among  teachers.  The  task  of  having  to  write 
curriculum  "exacerbated  the  imevenness:" 

"There's  always  strengths  and  weaknesses,  but  when  you  have  teachers  filling  dual 
roles  of  the  writers  of  the  curriculum  and  tire  implementers,  tiien  you  compound 
whatever  weaknesses  you  have.  And  also  compoimding  strengths,  that  happens  too." 

Another  teacher  reported  that  within  her  school,  some  of  her  colleagues  were  more 
engaged  in  design  implementation  than  others.  In  her  words,  "We  have  some  who  are  very 
reluctant,  who  are  barely  doing  anything...Now  slowly  but  surely  the)/ re  coming  aroimd,  but 
iti s  not  at  the  same  speed  as  a  lot  of  others."  Otiier  teachers  stated: 

have  to  have  your  commitment  factor.  Some  people  are  very  committed  to  it  and 
other  people  are  not,  so  that  affects  how  you're  going  to  implement  it." 
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"...as  far  as  some  elements  of  Co-NECT,  it's  pretty  basic  and  everybody  has  the  same 
level.  Now  to  the  extreme  of  using  the  internet  sites  or  all  those  types  of  things,  that 
would  depend  upon  that  individual  staff  member  or  grade  level  on  what  access  they're 
going  to  be  utilizing  out  there  that's  provided  by  Co-NECT.  So  I  think  there's  some  very 
beginning  stages  for  everyone  to  get  you  started  and  then  how  far  you  take  it  beyond 
that,  of  course,  is  how  much  time  or  energy  you  have  to  spend  in  Co-NECT." 

A  number  of  teachers  believed  that  NAS  designs  alone  did  little  to  help  children  who 
lacked  soUd  academic  foimdations.  Due  in  large  part  to  other  district  activities  that  were 
pushed,  some  came  to  view  designs  as  hands-on,  project  oriented  approaches  to  education  that 
built  on,  not  up,  basic  skills.  One  teacher  at  an  ELOB  school  expressed  that  her  students  needed 
more  structure.  She  stated  that  many  came  from  unstructured  home  environments  and  thus 
needed  more  orderly  classroom  experiences. 

"...it  would  work  probably  better  with  a  group  of  kids  that  are  on  grade  level,  that 
have  a  lot  of  self-control...If  they  come  from  a  home  where  there  is  no  structure,  (and) 
they  come  into  a  classroom  where  there  is  no  structure...that's  the  problem.  But  1  really 
feel,  and  I  might  be  wrong,  that  this  works  with  a  different  population  much  better  than 
what  it  has  worked  with  our  students." 

Teachers  at  a  Co-NECT  school  stated  that  their  design  units  had  to  be  "modified"  to  address 
their  students'  basic  skills  needs.  AtSFA/RW  and  MRSH  schools,  teachers  expressed  less 
doubt  about  the  potential  of  NAS  designs  to  bring  about  desired  change  in  school  achievement. 
This  may  have  to  do  with  the  fact  that  their  respective  designs  either  gave  them  a  curriculum  to 
follow  (SFA/RW)  or  topics  to  develop  and  standards  to  incorporate  (MRSH). 

In  the  next  chapter,  we  focus  specifically  on  the  curriculiun  and  instructional  conditions 
in  NAS  and  non-NAS  classrooms.  When  examining  tiiese  classroom  conditions,  it  is  important 
to  remember  the  influences  described  of  district,  design  teams,  and  the  school  environments  on 
NAS  teachers'  capacities  to  fully  implement  the  instructional  components  of  NAS  designs. 
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5.  CLASSROOMS  IMPLEMENTING  NAS  DESIGNS  IN  A 
REFORM-MINDED  DISTRICT 


As  discussed  in  a  previous  chapter,  the  district  introduced  NAS  designs  as  an 
overarching  initiative  for  improving  student  achievement.  Weary  of  tfie  piecemeal  practice  of 
reform  that  had  dominated  the  previous  administration's  efforts  at  school  improvement,  the 
superintendent  partnered  with  New  American  Schools  to  push  schools  to  comprehensively 
examine  change  from  within.  The  central  office  also  introduced  NAS  designs  to  its  schools 
with  hopes  that  each  would  sustain  the  reform  effort.  Simultaneously,  the  district  began  to 
critically  examine  the  curriculum  and  instructional  strategies  its  schools  employed. 

As  late  as  the  1998-1999  school  year,  no  school  had  yet  implemented  its  chosen  design 
completely  or  purely -that  is,  as  each  is  described  in  the  design  literature 
(http:/  / www.newamericanschooIs.org/teams:  Bodilly,  2001).  In  part  this  had  to  do  with  tire 
fact  that  most  schools  still  were  fairly  new  to  their  selected  designs.  In  this  chapter,  we  describe 
the  classroom  conditions  in  high-poverty,  urban  NAS  and  non-NAS  classrooms  to  gain  a  better 
imderstanding  of  the  progress  of  design  implementation  early  on  in  the  process.  Specifically, 
we  examine  the  overall  curriculum  changes  that  were  occurring  throughout  the  district  during 
the  course  of  this  study  and  then  describe  the  changes  in  classroom  organization  and 
instructional  practices. 

DISTRICT  RESTRUCTURING  OF  THE  CURRICULUM 

Because  students  within  the  San  Antonio  school  district  were  low-performers  on  TAAS, 
district  administrators  wanted  to  do  what  it  could  to  ensure  its  students'  success.  Believing  that 
teachers  had  neither  the  expertise  nor  the  time  to  learn  and  develop  a  sequential,  standards- 
aligned  curriculum  across  grade  levels,  the  district  took  on  this  role  full  force.  To  meet  the 
academic  needs  of  its  students,  the  district  not  only  adopted  particular  curricular  programs, 
over  the  years  it  also  introduced  its  schools  to  specific  instructional  strategies. 

By  the  1998-1999  school  year,  elementary  schools,  district  wide,  were  not  only  expected  to 
schedule  two  90-minute  blocks  of  iminterrupted  instructional  time  for  reading  and  math, 
respectively,  teachers  were  required  to  manage  time  witihin  these  blocks  in  prescribed  ways. 
Though  not  to  the  same  degree,  the  district  structured  language  arts  activities  (spelling, 
grammar,  and  writing)  as  well,  totaling  approximately  70  minutes  of  instruction  time  per  day. 
Thus,  roughly  three  hours  of  instructional  activities  were  mapped  out  for  all  the  district's 
elementary  school  teachers  to  follow  (SFA/RW  teachers  were  exempt  from  the  district  reading 
program). 
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Mathematics 

The  district  provided  the  most  detailed  guidance  with  respect  to  math.  Teachers  were 
given  a  pacing  guide— that  is,  a  highly  prescriptive  schedule  that  specified  which  lessons  were 
to  be  covered  during  a  given  week.  It  was  to  be  followed  conscientiously.  In  fact  district  staff 
or  school  administrators  periodically  looked  through  math  workbooks  to  ensure  accountability. 

"There's  more  of  a  lock  and  step  way  they  (the  district)  want  you  to  be...We  did  Everyday 
Math  last  year,  but  now  you  have  to  have  a  date  in  which  you  play  a  game,  and  you  have 
to  have  charts  on  the  (walls).  It  involves  a  lot  of  clerical  tasks...that  have  been  added, 
mandated...We  played  the  games  last  year,  but  now  I  have  to  go  back  there,  and  I  have  to 
put  a  date  when  I  played  the  games.  A  lot  of  clerical  things  like  that...They'll  come  in  and 
say,  'Well,  where's  your  math  message.'...It's  tihese  little  bitty  things  that  when  you  start 
to  think  of  all  the  things,  they  add  up.  Any  they  make  you  feel  as  if  you're  being 
restricted."  (Co-NECT) 

"...for  math  we  have  people  that  come  by.  They  would  come  in  to  check  up  in  our  math 
books  to  make  sure  we  were  on  a  certain  page...So  we  feel  like  if  we  miss  one  lesson  in 
math  and  they  come  in  to  check  and  we  haven't  done  it,  then  we'll  be  in  trouble.  So  that 
kind  of  rushes  us  with  the  math.  We  don't  get  to  teach  to  master  it." 

The  pacing  guide  was  a  source  of  great  frustration  to  many  teachers.  Teachers 
complained  that  for  fear  of  getting  behind,  they  felt  compelled  to  push  forward  regardless  of 
whether  or  not  their  students  understood  the  mathematical  concepts  covered  on  a  given  day. 
With  the  approach  of  TAAS,  teachers  reportedly  put  Everyday  Math  asiAe  to  focus  on  solving 
TAAS-formatted  matih  problems.  Because  the  district  required  that  Everyday  Alath\ovLmsis  be 
periodically  checked  for  completion,  teachers  foxmd  themselves  planning  "math-a-thons"  to 
catch  up  after  TAAS. 

"...we  have  TAAS  practice,  which  is  three  whole  days  we  lose  from  teaching  because 
we're  doing  the  test  Then  they  (the  district)  mandate  that  you  can't  get  behind,  so  we 
spend  the  rest  of  the  next  week  trying  to  double  up  on  our  math.  They  (district 
personnel)  come  in...  and  check  our  journals." 

Another  source  of  difficulty  for  teachers  was  that  Everyday  Math  assumed  a  certain  level 
of  backgroimd  math  knowledge  which  many  students  performing  below  grade  level  did  not 
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have.  The  teachers  expressed  that  the  program  failed  to  remedy  their  students'  lack  of  basic 
skills  or  vmderstanding  in  that  its  spiraling  approach  to  instruction  did  not  immediately  teach 
to  mastery.  Concepts  were  introduced  repeatedly,  in  greater  depth  each  time,  with  the 
understanding  that  eventually  s\xxdL&n\s  would  comprehend  the  material. 

"I  have  nine  students  who  are  learning  disabled.  And  I'm  supposed  to  teach  them  ...  a 

math  program  tiiat  they  may  not  be  ready  for  because  they  don't  have  the  foundation..." 

Given  that  at  many  schools,  older  students  were  introduced  to  Everyday  Math  not  ever 
having  been  exposed  to  its  terminology  and  strategies  in  earlier  grades,  teachers  initially 
experienced  difficulty  with  the  program.  After  being  in  place  for  three  years,  they  finally 
became  comfortable  enough  with  Everyday  Math,  making  instruction  easier. 

Ironically,  it  was  then  that  all  teachers  in  the  district  were  given  the  opportunity  to  officially 
adopt  this  math  program.  In  the  spring  of  1999,  teachers  voted  against  the  Everyday  Math  in 
favor  of  Advantage  Math,  a  math  program  more  traditional  in  nature. 

Finally,  because  math  instruction  tended  to  be  very  structured  and  time-consuming, 
teachers  treated  it  as  an  activity  in  and  of  itself.  That  is,  they  rarely  fotmd  ways  to  mesh  math 
instruction  with  their  design  projects  and/or  activities. 

"...Whenever  we  write  these  imits,  we  try  to  write  a  component  for  reading,  language, 
science,  social  studies,  everything...And  math  is  considered  separate.  In  fact,  we  don't 
even  have  to  write  a  math  component  into  the  unit."  (MRSH) 

"Math  has  nothing  to  do  with  Modem  Red."  (MRSH) 


"...I've  been  told  in  the  math,  I  have  to  spend  90  minutes  or  more  with  Everyday 
Math..so  having  to  do  that,  a  lot  of  times  I  just  haven't  always  put  the  math  component 
in  (the  expeditions),  because  I'm  going  to  have  to  teach  a  math  lesson  anyhow." 

"The  Everyday Math^N^  had,  we  were  on  a  time  schedule  to  finish  so  many  lessons  and 

it  didn't  matter  whether  that  lesson  went  along  with  something  we  were  doing  in 
expeditions  or  not." 

Reading  and  Language  Aits 

Though  less  prescriptive  in  nature  than  the  math  pacing  guide,  teaching  plans  for 
reading  and  language  arts  (called  "Instructional  Expectations  and  Learner  Outcomes")  were 
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issued  each  grading  period.  The  district  required  that  students  read  silently  in  class  for  twenty 
minutes  every  day.  Students  were  expected  to  maintain  a  book  log  detailing  their  readings  and 
thoughts  about  what  they  read.  The  teachers  also  were  required  to  engage  their  students  in 
daily  "focus  lessons"  during  which  time  they  were  to  introduce  reading  strategies  or  skills  (e.g., 
vocabulary,  comprehension,  text  analysis,  and  reading  strategies).  Finally,  teachers  were 
expected  to  divide  their  students  into  three  different  level  groups  and  hold  reading  circles  for 
twenty  minutes  each.  The  district  not  only  structured  the  time  involved,  it  also  suggested 
which  activities  be  made  available  to  students  left  on  their  own:  listening  centers,  seatwork 
targeting  specific  reading  skills,  and/ or  silent  reading. 

During  the  1998-1999  school  year,  after  having  brought  forth  its  instructional  plans  for 
math  and  reading,  the  Office  of  Curriculum  and  Instruction  introduced  its  teachers  to  a  detailed 
language  arts  guide.  Students  were  required  to  write  in  their  "read  and  respond  journals"  for 
several  minutes  every  day  after  listening  to  their  teachers  read  aloud.  Their  teachers  were  to 
provide  them  with  fifteen  minutes  of  daily  spelling  instruction.  Every  week,  students  were 
expected  to  memorize  lists  of  spelling  words  found  in  their  district-issued  textbook.  They  also 
were  required  to  work  on  grammar  for  ten  minutes  each  day  in  the  form  of  "Daily  Oral 
Language."  Towards  the  end  of  the  1998-1999  school  year,  the  district  strongly  encouraged 
classroom  teachers  at  all  grade  school  levels  to  incorporate  learning  centers  into  a  portion  of 
their  language  arts  block. 

"They  (the  district)  want  a  certain  way  that  we  teach  reading...We  now  have  a 
framework  that  says  do  10  minutes  of  this,  15  minutes  of  this,  15  minutes  of  that,  30 
minutes  of  this." 

"The  district  gave  us  a  very  strict  time  line  ffiis  year  [1998-1999].  Up  to  the  second 
almost.  Ten  minutes  here,  five  minutes  there,  and  you're  out  of  there...You  have  to  do 
just  15  minutes  silently,  ten  minutes  in  the  journals.. .And  tiiat's  really  affected  a  lot  of 
the  teaching." 

Writing  activities,  too,  were  structured  by  the  district.  Fourth  grade  teachers  foimd 
writing  an  especially  taxing  responsibility  in  light  of  the  fact  that  their  students  faced  the  TAAS 
writing  exam  for  the  first  time.  Although  students  at  all  grade  levels  were  expected  to  engage 
in  some  aspect  of  writing,  most  fourth  grade  teachers  felt  that  they  had  no  choice  but  to  teach 
writing  from  scratch,  given  that  these  skills  tended  to  be  tmderdeveloped.  In  addition  to 
introducing  and  practicing  the  process  of  writing  (from  brainstorming  to  drafting  to  editing),  it 
was  the  responsibility  of  fourth  grade  teachers  to  ensure  that  their  students  be  able  to  produce 
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works  written  in  different  styles.  Fourth  grade  students  were  expected  to  know  how  to  write 
classificatory,  narrative,  how-to,  and  persuasive  papers  not  only  for  their  own  good,  but  also 
for  TAAS.  The  district  dictated  the  types  of  mini  lessons  to  be  incorporated  into  daily  writing 
instruction  as  well  as  the  amount  of  time  to  be  spent.  Fourth  grade  teachers  found  though  that 
as  the  writing  TAAS  neared  (February),  thirty-five  minutes  every  day  was  too  little. 
Consequently,  during  the  winter  months,  they  devoted  hours  to  writing. 

The  district's  promotion  of  a  varied  assortment  of  instructional  practices  reflected  both 
the  back-to-basics  movement  as  well  as  more  progressive  approaches  to  teaching.  Some  of 
these  activities  actually  resembled  aspects  of  NAS  designs.  For  example,  the  structured  reading 
time,  ability-grouped  reading  circles,  and  read  and  response  journal  writings  reflected  elements 
of  SFA.  The  provision  of  computers  and  printers  in  all  classrooms,  as  well  as  the  district's 
request  that  all  teachers  receive  computer  training  and  incorporate  tiieir  newly  acquired  skills 
into  lesson  plans,  reminded  one  of  Co-NECT.  The  district's  emphasis  on  standards,  both  the 
state's  and  their  own,  resembled  MRSH.  Center-work  reflected  aspects  of  ELOB.  The  NAS 
schools  faced  competing  demands  between  the  district  initiatives  and  the  implementation  of 
NAS  designs.  As  one  teacher  stated: 

"...You're  told  what  to  do  and  how  to  do  it  for  reading.  You're  told  that  the  spelling  has 
to  be  done  everyday  according  to  a  spelling  list  that  has  nothing  to  do  with  whatever 
projects  your  students  are  working  on.  You're  told  what  page  and  lesson  to  be  on  in 
math  and  math  takes  an  hour  and  a  half...So  everything's  becoming  very  prescriptive. 
And  everything  has  to  tie  into  the  process  and  the  objectives  on  the  process." 

Given  the  district' s  focus  on  promoting  a  variety  of  instructional  approaches,  what  were 
the  instructional  conditions  in  NAS  classrooms?  Did  these  differ  from  those  in  non-NAS 
classrooms  within  a  reform-minded  district?  In  the  sections  that  follow,  we  draw  on  our 
different  sources  of  data  to  describe  the  early  implementation  of  NAS  designs  in  classrooms 
within  high-poverty  schools. 

CLASSROOM  ORGANIZATION 

Many  key  items  on  die  teacher  survey  focused  on  activities  at  the  classroom  level. 
Teachers  were  asked  about  skills  they  emphasized,  instructional  strategies  they  used,  grouping 
practices  within  classrooms,  assessments  and  grading  criteria,  instructional  materials  and 
technology.  Before  describing  other  instructional  strategies,  we  begin  with  a  view  of  average 
class  size  and  grouping  practices  and  then  go  on  to  examine  differences  found  between  NAS 
and  non-NAS  classrooms  across  a  variety  of  classroom  conditions  and  instructional  practices. 
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We  highlight  differences  across  1998  and  1999  for  our  longitudinal  sample  of  40  teachers  and 
bring  information  from  our  observations  and  interviews  where  appropriate. 

Class  Size 

An  important  educational  issue  has  been  the  class  size  in  which  students  receive 
instruction  (Stecher  and  Bornstedt,  2000;  2001;  Grissmer,  1999).  In  general,  teacher  survey 
responses  did  not  indicate  clear  distinctions  between  classrooms  in  NAS  and  non-NAS  schools 
in  terms  of  class  size.  Teachers  were  asked  to  indicate  the  number  of  students  in  their 
classrooms  at  the  time  they  responded  to  the  survey  (late  spring  of  1998  and  1999).  Overall, 
teachers  in  NAS  and  non-NAS  elementary  school  classrooms  had  about  18  students.  NAS 
teachers  reported  slightly  smaller  class  sizes  by  about  one  or  two  students  (Figure  5.1).  This 
overall  similarity  between  NAS  and  non-NAS  classrooms  is  likely  due  to  state  policy  on 
reducing  class  sizes  (Texas  Educational  Agency,  1999). 


Note:  Numbers  are  teacher-reported  and  based  on  a  total  sample  size  of  40  teachers— 

26  NAS  and  14  Non-NAS. 

Figure  5.1— Average  Class  Size  in  NAS  and  Non-NAS  Schools, 
Spring  1998  and  1999 


Grouping  Practices 

With  respect  to  student  grouping,  each  design  articulates  its  own  approach.  The  MRSH 
design  endorses  multiage,  multiyear  classroom  groupings  with  few  pullouts.  Within  tiiis 
environment,  students  can  be  observed  working  individually  as  well  as  in  a  variety  of  groups. 
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depending  on  project.  In  Co-NECT  schools,  one  also  expects  to  see  multiage,  multiyear 
arrangements.  However,  this  design  allows  for  other  grouping  strategies  as  well  -  tracking 
being  the  one  exception.  The  ELOB  design  also  does  not  endorse  tracking.  In  addition  to 
mainstreaming  special  education  students,  it  espouses  looping.  Teachers  are  to  stay  with  the 
same  students  for  two  to  three  years  to  nurture  relationships.  One  expects  to  see  ELOB 
students  working  in  small  groups.  The  SFA/RW  design,  unlike  the  other  designs,  openly 
endorses  the  homogenous  grouping  of  students  by  reading  level.  However,  students  are  to  be 
assessed  every  eight  weeks  and  reassigned  as  appropriate,  providing  a  flexible  use  of  grouping 
for  instructional  purposes.  This  way,  students  can  receive  the  individual  attention  they  need. 

Teachers  in  NAS  classrooms  tended  to  report  using  non-traditional  grouping  practices, 
although  for  the  entire  sample  such  practices  were  not  widespread  throughout  any  of  the 
schools.  Mean  responses  on  survey  items  that  addressed  this  issue  (using  6-point  scale  varying 
from  does  not  describe  my  school  to  clearly  describes  my  school)  are  provided  in  Table  5.1.  By 
1999,  when  considering  whether  grouping  was  flexible  or  was  organized  into  block  scheduling, 
NAS  teachers  reported  tended  to  report  in  the  middle  of  tite  6-point  scale  (2.7  and  2.9, 
respectively).  Compared  with  NAS  teachers'  reports  on  these  indicators,  non-NAS  teachers 
scored  somewhat  lower.  In  addition,  NAS  teachers  reported  an  increase  between  1998  and  1999 
when  asked  about  whether  their  schools  used  more  traditional  ability  grouping  on  a  regular 
basis. 


Table  5.1 

Teacher  Reported  Means  on  the  Extent  to  Which  Grouping  Practices  Characterize  NAS  and 

non-NAS  Schools,  Spring  1998  and  1999 


1998 

1999 

NAS 

Non- 

NAS 

NAS 

Non- 

NAS 

Student  grouping  is  fluid,  multi-age,  or  multi-year 

3.1 

2.2 

2,7 

2.0 

Students  are  organized  into  instructional  groups  using 
block  scheduling  for  specific  curricular  purposes 

3.5 

2.4 

2.9 

2.6 

Students  in  this  school  are  grouped  by  achievement 
levels  into  high,  middle,  and/  or  remedial  instructional 
groups  on  a  regular  basis 

2.8 

2.3 

3.5 

2.7 

Note:  Means  are  based  on  teacher  reports  in  a  total  sample  size  of  40  teachers  — 

26  NAS  and  14  Non-NAS. 

Responses  rage  from  1  ''does  not  describe  my  school"  to  6  "clearly  describes  my  school" 
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Our  observations  in  NAS  and  non-NAS  schools  revealed  that  not  all  of  the  San  Antonio 
NAS  schools  implemented  the  design  team  approaches  to  student  grouping.  And  in  fact, 
variation  was  observed  within  design  by  school  as  well.  The  SFA/RW  schools  strictly  followed 
the  guidelines  laid  out  for  them.  They  really  had  no  choice,  since  the  implementation  of 
Success  for  AU  involved  very  specific  actions.  There  was  no  room  to  deviate  given  the  scripted 
curriculum  and  accompanying  materials.  At  the  two  SFA/RW  schools  we  followed,  the 
reading  assessments,  administered  approximately  every  eight  weeks,  determined  how  students 
should  be  grouped.  In  most  cases,  students  tended  to  be  similar  in  age.  The  teachers  employed 
all  the  instructional  strategies  called  for  in  their  SFA  lesson  plans.  Grouping  strategies  Vcuied 
by  activity.  Students  not  only  learned  as  a  class,  they  also  worked  individually  and  in  small 
groups. 

Looping  was  practiced  at  the  teachers  we  observed.  The  gifted  and  talented  students 
tended  to  be  grouped  together,  however.  The  ELOB  students  we  observed  were  given 
opportunities  to  work  both  independently  and  cooperatively.  When  asked  to  work  together, 
students  were  given  the  option  to  work  alone,  which  some  took  advantage  of.  However,  when 
asked  to  work  independently,  students  had  no  say.  Relative  to  students  exposed  to  SFA/RW, 
the  ELOB  students  appeared  to  be  less  familiar  with  cooperative  learning.  When  given  the 
chance  to  work  collaboratively,  the  students  rarely  solved  problems  togetiier.  They  tended  to 
work  independently.  Often,  these  students  were  put  in  small  groups  to  assist  or  inspire  one 
another  as  they  worked  on  individual  projects. 

The  grouping  practices  of  the  two  Co-NECT  schools  we  observed  differed  from  one 
another.  One  engaged  in  looping;  one  did  not.  In  the  school  that  practiced  looping,  the  fourth 
and  fifth  grade  teachers  collaborated  quite  extensively.  However,  they  did  not  instruct  each 
other's  students.  Desk  arrangements  varied  by  classroom.  Students  were  observed  working 
both  individually  and  in  small  groups. 

As  for  the  MRSH  schools  we  followed  in  San  Antonio,  no  evidence  of  performance 
grouping,  multi-age  classrooms,  multi-grade  teaching  teams  was  observed.  Looping,  however, 
was  practiced  at  one  school.  Students  were  not  tracked  by  ability,  but  teachers  did  report  that 
the  gifted  and  talented  students  tended  to  be  the  responsibility  of  one  teacher.  The  teachers  at 
both  MRSH  schools  periodically  regrouped  their  students  and  rearranged  desks  as  appropriate, 
but  such  changes  tended  to  be  socially  based.  Students  were  grouped  differently  as  necessary 
throughout  the  day,  but  tire  groupings  did  not  vary  that  dramatically.  Students  worked  either 
independently  or  in  small  groups.  They  also  learned  as  a  whole  class. 

When  it  came  time  to  prep  hard  for  TAAS,  teachers  at  several  design  schools  worked 
with  a  variety  of  students.  That  is,  grade  level  teachers  pooled  their  class  lists 
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and  dien  divided  all  students  by  skill  level  and/or  ability.  Each  grade  level  teacher  worked 
with  one  ability  group  for  a  set  amount  of  time. 

INSTRUCTIONAL  PRACTICES 

Instructional  Tasks 

We  examined  a  variety  of  survey  questions  about  student  skills  and  the  extent  to  which 
the  NAS  teachers  emphasized  these  skills  by  asking  how  often  students  were  expected  to 
demonstrate  them.  Since  NAS  designs  tend  to  emphasize  higher-order  thinking  skills  over 
memorization  and  drilling,  we  might  expect  teachers  in  NAS  classrooms  to  report  lower  levels 
of  memorization  and  higher  levels  of  other  types  of  critical  thinking  skills  (Bodilly,  2001). 
Figure  5.2  shows  mean  responses  on  a  four-point  scale  (almost  never  to  every  lesson)  in  which 
teachers  were  asked,  "How  often  do  you  have  students  memorize  facts  or  problems?" 
Memorization  tended  to  be  emphasized  more  by  non-NAS  teachers,  but  only  in  1998.  The 
slight  increase  in  NAS  responses  in  1999  may  be  due  to  the  increased  pressures  schools  were 
experiencing  to  switch  to  "skill  and  drill"  techniques  to  prepare  for  the  TAAS. 


Note:  Means  are  based  on  teacher  reports  in  a  total  sample  size  of  40  teachers  — 

26  NAS  and  14  Non-NAS. 

Figure  5.2— Average  Teacher  Response  for  Having  Students  Memorize  Facts  or  Problems  in 
the  T3^ical  Lesson  in  NAS  and  Non-NAS  Schools,  Spring  1998  and  1999 
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Other  survey  items  measured  how  often  students  were  requested  to  demonstrate 
analytical  and  higher-order  thinking  skills.  We  found  few  differences  in  NAS  teachers' 
responses  compared  with  non-NAS  teachers  when  asked  how  often  students  use  library 
sources,  brainstorm  ideas  for  written  work,  debate  ideas,  apply  concepts  or  skills  from  earlier 
lessons,  judge  and  critique  their  own  and  each  others'  work,  reflect,  and  work  m  teams  toward 
a  common  goal. 

On  items  asking  whetirer  students  "independently  conduct  and  design  their  own 
research  project,"  "relate  the  material  to  their  life  or  their  commimity,"  and  "draft  or  re-draft 
work,"  non-NAS  teachers  reported,  on  average,  a  substantially  higher  degree  of  emphasis  in 
both  1998  and  1999. 

We  also  asked  teachers  about  the  types  of  instructional  strategies  they  employed  in  their 
classrooms.  We  sorted  their  responses  according  to  more  conventional  or  reform-like 
categories  of  instruction. 

Conventional  Instructional  Practices 

We  asked  teachers  to  indicate  how  often  they  used  particular  instructional  strategies  in 
their  classes,  using  a  five-point  scale,  ranging  from  never  to  almost  every  day.  Responses  from 
teachers  in  NAS  and  non-NAS  schools  varied  only  slightly  in  both  years  when  it  came  to 
reporting  on  conventional  instructional  strategies  such  as: 

•  Having  students  respond  orally  to  questions  on  subject  matter; 

•  Work  individually  on  written  assignments/ worksheets  in  class; 

•  Practice  or  drfll  on  computational  skills; 

•  Read  textbooks  or  supplementary  materials;  and 

•  Work  on  next  day's  homework  in  class. 

With  the  exception  of  the  last  item,  well  over  90  percent  of  all  40  teachers  reported  using  these 
strategies  at  least  once  or  twice  a  week.  Between  21  and  29  percent  of  teachers  indicated  having 
students  work  on  their  next  day's  homework  in  classes  that  often. 

In  general,  teachers  in  the  NAS  schools  indicated  less  reliance  on  more  conventional 
instructional  strategies  than  teachers  in  non-NAS  schools.  Teachers  in  non-NAS  schools  were 
much  more  likely  to  use  conventional  instructional  strategies  such  as  lecturing,  administering  a 
test  over  a  full  class  period,  and  administering  quizzes. 

Ref  onn-like  Instructional  Practices 

Instructional  strategies  classified  as  reform-like  were  used  to  construct  a  reform 
composite.  Responses  from  two  scales  were  standardized  —  to  indicate  (1)  how  often  teachers 
used  the  instructional  strategies  witih  this  class  (a  5-point  scale  ranging  from  never  to  almost 
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every  day)  and  (2)  how  often  teachers  had  students  demonstrate  skills  (a  4-point  scale  ranging 
from  almost  never  to  every  lesson).  The  following  items  were  included  in  the  reform 
composite:!^ 

•  Have  students  listen  to  an  outside  speaker/ expert; 

•  Have  students  perform  research  projects; 

•  Use  manipulatives  to  demonstrate  a  concept; 

•  Have  students  work  with  manipulatives; 

•  Have  small  groups  work  on  problems  to  find  a  joint  solution; 

•  Have  the  whole  class  discuss  solutions  developed  in  small  groups; 

•  Have  students  work  on  problems  for  which  there  is  no  obvious  method  of  solution; 

•  Have  students  represent  and  analyze  relationships  using  tables  and  graphs; 

•  Have  students  respond  to  questiojis  or  assignments  that  require  writing  at  least  a 
paragraph; 

•  Have  students  keep  a  journal; 

•  Summarize  main  points  of  today's  lesson; 

•  Have  students  work  on  projects  in  class; 

•  Have  students  explain  their  reasoning; 

•  Have  students  represent  and  analyze  relationships  using  tables,  graphs,  or  charts;  and 

•  Have  students  work  on  problems  for  which  there  is  no  immediately  obvious  method 
or  solution. 

Teachers'  responses  for  this  reform-like  instructional  composite  are  provided  in  Figure 
5.3.  While  the  average  use  of  reform-like  instructional  practices  increased  for  NAS  and  non- 
NAS  teachers  between  1998  and  1999,  teachers  in  NAS  schools  reported  higher  levels  than  their 
counterparts  in  non-NAS  schools.  For  example,  in  1999  54  percent  of  NAS  teachers  reported 
using  practices  in  the  reform-Uke  composite  at  least  once  or  twice  a  week  compared  with  36 
percent  of  non-NAS  teachers. 


The  alpha  reliability  for  this  composite  was  .77  for  both  1998  and  1999.  The  range  of  correlations  for  the 
individual  items  was  .17  to  .20  in  both  years. 
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Note:  Percentages  are  based  on  a  total  sample  size  of  40  teachers— 26  NAS  and  14  Non-NAS. 

Figuie  5.3— Percentage  o£  Teachers  Who  Reported  Using  Reform-like  Instructional  Practices 

at  Least  Once  or  Twice  a  Week 
in  NAS  and  Non-NAS  Schools,  Spring  1998  and  1999 


Given  the  unique  features  of  designs  and  their  respective  emphases  on  student  work 
products,  one  would  reasonably  expect  to  see  differences  in  classroom  appearance,  set  up,  and 
student  work  displays  across  design  schools.  We  found  that  the  level  of  impact  that  designs 
had  upon  classroom  appearances  changed  from  the  first  year  of  our  study  to  the  second.  In 
the  first  year,  design  elements  were  often  clearly  identifiable.  In  MRSH  classrooms,  standards 
were  posted  next  to  student  work.  Word  walls  and  team  score  sheets  were  posted  in  SFA/RW 
classrooms.  Rich  classroom  libraries  were  found  in  Co-NECT  classrooms  and  student  work 
linked  to  themes  and  a  multi-disciplinary  perspective  was  posted  in  hallways  and  classrooms. 
Displayed  tfiroughout  BLOB  classrooms  were  expedition  themes,  student-developed  rubrics, 
and  drafts  and  re-drafts  of  student  writing. 

In  the  second  year,  the  influence  of  the  central  office  was  more  obvious  in  that  across  our 
sample  schools,  identical  posters  outlining  the  writing  and  reading  processes,  math  definitions, 
and  district-developed  rubrics  were  most  commonly  found  taped  to  classroom  walls.  In  every 
classroom,  word  walls  were  found  as  well  as  postings  of  student  work  on  bulletin  boards.  The 
district  provided  all  classrooms  with  six  computers  and  at  least  one  printer.  All  computers 
were  loaded  with  the  same  programs.  The  same  trade  books  were  found  in  every  room.  In 
most  classrooms,  desks  were  commonly  arranged  in  clusters  of  four  to  six.  Teachers  across 
schools  reportedly  rearranged  students  quite  regularly  to  enable  classmates  to  get  to  know  one 
another. 
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One  could  teU  that  classrooms  were  part  of  given  designs  only  because  teachers 
advertised  this  fact  through  posters.  In  MRSH  classrooms,  various  standards  tended  to  be 
posted  on  bulletin  boards  next  to  displays  of  student  work.  In  ELOB  classrooms  design 
principals  were  often  found  taped  to  walls.  Co-NECT  classrooms  tended  to  be  less  distinctively 
marked.  The  selection  of  student  work  on  display  as  well  as  reading-related  posters  clearly 
distinguished  SFA/RW  classrooms  from  the  rest.  The  appearance  of  classrooms  as  well  as  even 
the  work  displayed  revealed  teachers'  efforts  to  comply  with  both  die  districts'  demands  and 
those  of  their  selected  designs. 

Classroom  observations  revealed  a  schism  with  respect  to  design  implementation.  The 
designs  per  se  were  not  the  source  of  teachers'  problems.  The  difficulties  arose  out  of  the 
struggle  to  merge  district  demands  with  design  practices  while  maintaining  the  integrity  of 
designs.  The  focus  in  NAS  schools  was  not  how  well  designs  were  being  implemented  and 
affecting  the  educational  climate,  but  rather  how  well  teachers  were  preparing  for  TAAS  and 
following  district  prescriptions.  The  teachers  were  left  on  their  own  to  figure  out  how  to 
incorporate  district  initiatives  into  their  lesson  plans  in  the  spirit  of  their  designs.  To  determine 
whether  NAS  teachers  and  students  actually  interacted  witii  each  other  and  subject  materials  in 
ways  reflective  of  design  teams'  curricular  and  instructional  theories,  classroom  activities  were 
examined  with  care. 

Use  of  Assessments 

NAS'  ultimate  goal  is  to  help  schools  realize  improved  student  performance.  In  San 
Antonio,  the  TAAS  is  the  preeminent  measure  of  student  performance.  Clearly,  it  was  very 
important  to  the  district  and  its  schools  that  students  perform  respectably  on  this  standardized 
test.  Like  many  urban  school  districts,  the  San  Antonio  schools  functioned  in  a  high-stakes 
testing  environment.  Given  this  climate,  the  district  assumed  much  of  the  responsibility  for 
assuring  the  accountability  of  its  schools  and  teachers  by  imposing  a  set  of  instructional 
standards  and  practices  across  the  board —regardless  of  the  existence  of  NAS  designs  at  its 
schools.  Moreover,  teachers  employed  "skiU  and  driU"  in  their  efforts  to  prepare  students  for 
TAAS. 

Teachers  from  different  design  schools  consistently  remarked  that  with  the  approach  of 
TAAS,  instruction  narrowed  to  honing  tested  skills.  They  questioned  whether  the  instructional 
approaches  espoused  by  their  respective  designs  (e.g.,  project-based  and  interdisciplinary) 
alone  could  bring  about  test  score  increases.  In  the  words  of  one  teacher  whose  school  dropped 
the  design  at  the  start  of  what  would  have  been  its  third  year,  "There  are  some  faculty  members 
who  bemoan  the  loss  of  their  love  [the  design].  But  I  think  all  of  us  saw  that  it  wasn't  getting  us 
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where  we  needed  to  be  when  the  requirement  here  is  having  a  certain  level  of  performance  on 
the  TAAS  test." 

As  described  in  the  previous  chapter,  teachers  reported  that  TAAS  preparation  was  a 
matter  of  practice  and  familiarizing  students  with  test  format.  Given  the  district s  numbers  of 
students  performing  below  grade  level,  it  was  important  to  them  that  they  directly  address 
tested  skills.  Teaching  students  how  to  take  the  TAAS  was  just  as  important  to  teachers  as 
reviewing  the  skills  to  be  tested.  Such  an  assessment  system  within  a  high-stakes 
accountability  environment  does  not  fit  well  witii  most  of  the  design  teams'  approaches  (except 
SFA/RW),  even  though  each  team  expresses  a  desire  to  work  with  whatever  local  assessment  is 
in  place  (Bodilly,  2001). 

No  design  was  developed  solely  to  produce  high  standardized  test  scores.  Each  design  in 
fact  promotes  its  own  methods  of  assessment.  The  ELOB  design  calls  for  authentic  assessment, 
including  performance-based  exhibitions,  student  portfolios,  and  student  self-assessments. 
Student  work  products  are  to  be  shared  with  the  community  to  ensure  a  certain  quality  of  work 
and  to  encourage  support.  The  Co-NECT  design  relies  on  a  mix  of  assessments  and  portfolios 
to  judge  student  achievement  Student  assessments,  guided  by  exemplar  products  and  rubrics, 
are  to  encourage  the  process  of  setting  goals  and  measuring  progress.  The  MRSH  design,  like 
Co-NECT,  also  relies  on  various  assessments.  These  include  MRSH  developed  tests,  watershed 
assessments,  and  embedded  assessments.  The  SFA/RW  design  team  endorses  a  performance- 
based  approach  to  assessment  Included  are  hands-on  demonstrations  and  portfolios. 

Teachers  across  the  various  NAS  schools  reported  using  a  variety  of  assessment 
measures.  These  ranged  from  observations  to  student  work  assessments  to  objective  paper  and 
pencil  tests.  All  teachers  made  some  use  of  rubrics.  The  district  encouraged  this  and  even 
provided  teachers  with  several  rubrics  to  use.  In  compliance 

with  district  policy,  teachers  also  regularly  administered  spelling,  math,  and  reading  tests,  as 
weU  as  year-end  reading  and  math  tests. 

The  teachers  at  the  MRSH  schools  we  observed  used  rubrics  to  assess  their  students' 
projects.  One  teacher  developed  multiple  choice  tests  to  assess  her  students'  knowledge.  At 
both  schools,  methods  of  assessment  varied  by  subject.  At  neither  school  was  there  mention  of 
watershed  assessments  or  individual  education  compacts. 

The  Co-NECT  teachers  we  spoke  with  also  used  district-established  rubrics  to  some 
extent  None  felt  entirely  comfortable  with  the  process,  however.  With  respect  to  Co-NECT 
projects,  die  teachers  seemed  to  apply  different  criteria  depending  on  assignment  type.  There 
appeared  to  be  no  obvious  measures  of  assessment  in  place.  The  Co-NECT  teachers  we 
observed  did  not  put  as  much  time  into  developing  assessment  tools  as  they  did  into  planning 
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units.  Students  were  rarely  shown  exemplary  work.  Nor  was  the  grading  criteria  explained  to 
them  up  front. 

At  the  ELOB  schools  we  observed,  the  practice  of  assessment  appeared  to  change 
greatly  from  the  spring  of  1998  to  the  spring  of  1999.  In  the  first  year,  students  developed  and 
applied  their  own  rubrics  and  re-drafted  their  work.  But,  in  the  second  year,  teachers  and 
students  were  not  seen  reexamining  graded  assignments  or  discussing  ways  to  improve  upon 
them.  Nor  did  there  appear  to  be  any  evidence  of  reflection  on  the  quality  of  student  work. 
Rubrics  were  used  at  times  to  grade  student  work,  but  it  is  not  clear  how  consistently.  The 
rubrics  used  by  ELOB  teachers  tended  to  be  those  developed  by  other  outside  experts.  The 
teachers  at  both  ELOB  schools  we  followed  mentioned  that  they  really  were  rmclear  as  to  how 
their  design  defined  good  quality  work. 

"Show  us  what  a  good  product  is.  Is  this  a  good  product?  Give  us  an  (example  of) 

authentic  product.  Is  a  book  an  authentic  product?.. .What  is  an  authentic  product? 

What  does  that  look  like?" 

MRSH  and  Co-NECT  teachers  expressed  the  same  uncertainty  about  their  products: 

"If  I  can  see  it,  see  a  product...then  I  get  an  idea.  But  just  hearing  it,  I  can't  picture  that." 

(Co-NECT) 

"I  would  like  to  see  how  does  it  look  whenever  a  good  unit  is  being  taught."  (MRSH) 

The  SFA/RW  teachers  we  spoke  with  were  only  familiar  with  SFA.  Given  its  scripted 
approach  to  reading,  the  practice  of  assessment  was  not  an  issue  for  SFA/RW  teachers.  At  both 
schools,  reading  assessments  were  administered  about  every  eight  weeks  or  so  to 
determine  students'  reading  levels.  Based  on  Iheir  results,  students  were  regrouped  as 
necessary.  Reading  teachers  used  team  score  sheets  to  record  their  students'  grades  on  SFA 
assigrunents.  The  score  sheet  enabled  them  to  chart  their  students'  progress  over  time. 

On  the  teacher  survey,  respondents  were  also  asked  to  indicate  the  importance  of  varying 
types  of  assessment  instruments  using  a  4-point  scale  (not  very  important  to  very  important). 
The  only  assessment  which  teachers  in  NAS  schools  rated  as  being  important  was  the  use  of 
student  portfolios.  In  general,  non-NAS  teachers  tended  to  indicate  higher  levels  of  importance 
on  most  assessments  used,  including  multiple  choice  and  essay  tests  (whether  developed  by  the 
teacher,  an  outside  source,  or  found  within  a  unit  or  book),  completion  of  homework,  student 
work,  open-ended  problems,  individual  projects/ report,  performance  tasks  or  events,  and 
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standardized  test  results.  Teachers  in  botir  NAS  and  non-NAS  schools  responded  similarly 
with  respect  to  the  importance  of  assessing  student  participation  in  class  and  group 
projects/  reports. 

Use  of  Instructional  Materials 

Little  variation  was  foimd  between  NAS  and  non-NAS  teachers  in  the  use  of  instructional 
materials.  Both  groups  of  teachers  tended  to  use  a  variety  of  materials  fairly  frequently  in  their 
classrooms,  including  textbooks,  literature  books,  workbooks,  computers,  calculators, 
manipulatives,  audio-visual  equipment,  games,  lab  equipment,  and  library  materials.  Teachers 
in  non-NAS  schools,  however,  indicated  slightly  higher  levels  of  satisfaction  with  the 
availability  of  instructional  materials  in  their  classrooms. 

NAS  teachers  were  more  likely  than  non-NAS  teachers  to  report  that  inadequate 
instructional  materials  hindered  student  achievement  (see  Figure  5.4).  For  example,  in  1998  35 
percent  of  NAS  teachers  reported  that  instructional  materials  hindered  students'  academic 
success  either  moderately  or  greatly  compared  with  21  percent  of  non-NAS  teachers.  For  these 
same  teachers,  16  percent  of  NAS  teachers  and  7  percent  of  non-NAS  teachers  reported  that 
inadequate  instructional  materials  posed  a  barrier  to  students'  academic  success  in  1999.  It  is 
likely  that  this  overall  decline  was  due  to  the  extensive  curricular  programs  (mathematics, 
reading,  language  arts)  being  implemented  district-wide. 


■  NAS 
□  Non-NAS 


Note:  Percentages  are  based  on  a  total  sample  size  of  40  teachers— 26  NAS  and  14  Non-NAS. 

Figure  5.4— Percentage  of  Teachers  Who  Reported  that  Inadequate  Instructional  Materials 
Moderately  or  Greatly  Hindered  Students'  Academic  Success 
in  NAS  and  Non-NAS  Schools,  Spring  1998  and  1999 
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In  the  first  year  of  our  study,  our  classroom  observations  revealed  that  the  use  of 
instructional  materials  varied  greatly  among  the  NAS  designs.  While  non-NAS  and  MRSH 
students  were  seen  using  textbooks,  traditional  literature,  and  pencil  and  paper,  students  in  Co- 
NECT  and  ELOB  schools  were  observed  accessing  various  other  materials  during  project  time, 
including  reference  books,  the  internet,  and  sometimes  multi-media  software.  They  also  left 
the  room  to  gather  resources  from  the  library  or  other  classrooms.  During  the  SFA  block, 
SFA/RW  students  used  the  SFA  workbooks  and  trade  books  supplied  by  the  design. 

By  the  second  year,  materials  used  for  instructional  purposes  tended  to  be  more  similar 
across  schools,  regardless  of  design  presence.  The  technology  available  in  classrooms  was 
similar,  for  example.  Students  across  the  district  read  many  of  the  same  books  and  engaged  in 
tasks  that  required  little  more  than  pencil  and  paper.  Even  the  SFA  classrooms  contained  the 
same  reading  materials  as  the  other  district  schools,  although  they  were  not  used  during  the 
reading  block.  Given  that  the  district  required  all  schools  to  adopt  the  same  matir  curriculum 
and  employ  specific  instructional  strategies  in  the  areas  of  reading  and  language  arts,  it  is  not 
all  that  surprising  that  the  materials  found  in  classrooms  tended  to  be  more  alike  tiran  different. 
However,  although  the  materials  were  similar,  they  were  not  always  utilized  in  the  same  way 
by  all  campuses.  For  example,  one  of  our  Co-NECT  schools  employed  the  available  technology 
more  readily  than  most  other  campuses. 

EXAMPLES  OF  STUDENT  WORK 

The  work  students  produced  also  tended  to  look  alike  across  schools.  However,  a  close 
examination  of  the  work  collected  revealed  that  some  teachers  more  so  than  others  espoused  an 
interdisciplinary,  reform-minded  instructional  approach 

So  that  RAND  could  better  understand  the  nature  of  student  work  assigned  over  the 
course  of  one  year,  teachers  were  asked  to  submit  examples  of  student  work  every  three 
months  or  so.  No  criteria  were  established  with  regard  to  work  submissions.  The  teachers 
simply  were  asked  to  provide  examples  of  t5q)ical  work  assignments  produced  by  several  of 

on 

their  randomly  selected  students. 

Quite  possibly  the  submitted  work  was  not  entirely  representative  of  all  student 
assignments  made  by  a  given  teacher;  nonetheless,  the  student  pieces  did  provide  a  flavor  of 
the  types  of  activities  assigned  by  each  of  the  teachers  in  our  sample.  Overall,  the  student  work 
revealed  the  use  of  a  imiform  fourth  grade  curriculum  across  schools  within  San  Antonio.  For 

RAND  randomly  selected  a  quarter  of  students  in  each  class  every  three  months.  Once  a  student  was 
selected,  his  or  her  name  was  removed  from  the  class  roster. 
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example,  in  social  studies,  students  studied  Texas  history,  including  such  topics  as  regions, 
Indians,  exploration,  missions,  settlements,  and  tall  tales. 

Although  Ihe  submissions  were  similar  in  content  and  skills  emphasized,  they  revealed 
that  teachers  did  not  present  the  curriculum  in  the  same  ways.  For  example,  some  teachers 
made  efforts  to  cover  multiple  objectives  simultaneously  by  having  students  write  biographies 
about  famous  people  from  Texas  history.  Other  teachers  allowed  students  to  write  biographies 
on  any  famous  person,  past  or  present. 

NAS  teachers  tended  to  incorporate  the  practice  of  reading  and  writing  skills  whenever 
they  could.  Thus,  when  covering  a  social  studies  topic,  or  on  rare  occasions  a  science  lesson,  for 
example,  they  had  their  students  activate  the  reading  process  and  write  how-to,  classificatory, 
persuasive,  and/or  narrative  papers.  By  contrast,  non-NAS  teachers  tended  to  use  more 
traditional  and  compartmentalized  strategies  to  teach  social  studies  and  science,  such  as  fill-in 
the  blank  questions  at  the  end  of  a  social  studies  chapter  (see  Tables  5.2  and  5.3  for  examples  of 
lesson  plans). 

Although  design  teachers  made  efforts  to  integrate  district  or  state  required  content 
material  into  their  units  of  study,  they  did  not  consistently  engage  their  students  in 
interdisciplinary  work.  Many  tended  to  endorse  a  more  traditional  instructional  style  some  of 
the  time,  teaching  subjects  as  discrete  disciplines  and  rel5dng  on  worksheets  and  textbook 
assignments.  In  some  ways,  they  were  encouraged  to  do  so.  For  example,  teachers  were 
required  to  calculate  report  card  grades  for  each  subject;  moreover,  they  were  expected  to 
follow  state  and  district-mandated  curricular  and  instructional  guidelines  by  subject.  Teachers 
had  the  latitude  to  tie  subjects  together,  but  the  constraints  within  which  they  operated  tended 
to  limit  their  vision. 

As  revealed  earlier,  teachers  across  design  schools  reported  that  they  found  it  difficult  to 
integrate  Everyday  Math  lessons  witii  other  subjects.  Thus,  they  taught  it  as  a  subject  in  and  of 
itself  and  our  analysis  of  student  work  was  consistent  with  the  teacher  challenges  expressed 
during  interviews  and  observations. 

In  general,  die  SFA/RW  and  non-NAS  schools  modeled  a  traditional  approach  to 
instruction,  focusing  on  compartmentalized  skill  exercises.  They  utilized  worksheets  and 
textbook  assignments  as  prescribed  by  SFA.  The  student  work  submitted  by  ELOB  and  Co- 
NECT  teachers  reflected  a  more  interdisciplinary  unit-based  approach  to  instruction  and 
showed  more  evidence  of  contemporary  teaching  practices  (e.g.,  the  use  of  rubrics,  cooperative 
grouping,  and  integrated  technology).  One  of  our  Co-NECT  schools,  more  so  than  the  other, 
however,  reflected  a  more  traditional  approach  to  teaching.  The  MRSH  schools  employed  a 
mix  of  strategies,  including  project-oriented  work  that  incorporated  a  variety  of  skills  in  the 


process. 
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Table  5.2 

Example  of  a  Conventional  Lesson  Flan 


Language  Arts 

Math 

Social  Studies 

Science 

T  A  AS  Master  R  38 
Write  a  story  about 
the  day  the 

Snowman  came  to  life, 
listing  ideas 

Give  Spelling  pre-test 
on  unit  15 

Vocabulary  focus.  Beat 
the  Story  Drum,  p.  227 

Test  Ready  33  &  34 

Independent  Reading, 
p.  227-247  using  the 
reading  process  &  do 
Sum  It  Up 

Students  will  locate 
places  on  an  atlas 
map,  solve  problems 
in  math  box  51,  and 
record  information 
about  latitude  and 
longitude.  Everyday 
Math  Workbooks  p. 
236-238 

Student  journal  p.  145 
&146 

Study  Links  50  &  51 
World  Tour  Guide  Book 
and  record  sheets  1, 5 
-7 

A  Paradise  Called  Texas 
-  Use  Reading  Process 
to  gather  information 
about  the  German 
culture  in  Texas  for  a 
report  to  be  presented 
in  class 

Students  will  make 
and  use  a  book  about 
magnetism  -  AIMS 
Lesson.  After  creating 
the  book,  reading  and 
discussing  info  on 
magnets,  students  will 
experiment  with 
magnets. 

Table  5.3 

Example  of  a  Lesson  Plan  Consistent  with  Reform-Like  Orientation  of  a  NAS  Design 

Title:  Europe  in  the  Middle  Ages  ~ 

Abstract  This  xmit  focuses  on  the  development  of  civilization  during  the  European  Middle  Ages 
from  400  A.D.  to  1500  A.D.  The  social  studies  component  is  research-based  and  includes 
geography,  feudalism,  chivalry,  timelines,  and  the  study  of  important  people.  The  language  arts 
focus  is  on  writing  and  dialect  Math  includes  measurement  and  construction  of  castles.  Reading 
incorporates  myths,  legends,  and  an  understanding  of  real  and  fictional  characters.  Science  is 
included  wifli  die  study  of  hygiene  and  diseases.  Art  and  drama  are  fully  executed  throughout 
the  entire  unit  with  die  production  of  a  play  and  the  creation  of  artifacts. 

MRSH  Core  Knowledge  Sequence:  World  Civilization  for  tiie  fourtii  grade. 

Note:  the  teacher  goes  on  to  outiine  12  lessons  in  the  unit 
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Use  of  Technology  for  Instructional  Purposes 

Since  NAS  designs  such  as  Co-NECT  and  MRSH  tend  to  emphasize  the  use  of  technology 
in  the  classroom,  two  survey  items  addressed  this  issue.  As  shown  in  Figure  5.5,  compared  with 
NAS  teachers,  those  in  non-NAS  schools  indicated  slightly  higher  levels  of  technology  being 
used  as  an  integrated  classroom  resource.  In  both  sets  of  schools  teachers  reported  that 
technology  use  declined  from  1998  to  1999,  despite  a  district-wide  effort  to  support  the  use  of 
computers  in  all  elementary  schools. 


■  NAS 
□  Non-NAS 


Note:  Percentages  are  based  on  a  total  sample  size  of  40  teachers— 26  NAS  and  14  Non-NAS. 

Figure  5.5 -Percentage  of  Teachers  Who  Reported  that  Technology  was  Clearly  an  Integrated 
Classroom  Resource  in  NAS  and  Non-NAS  Schools,  Spring  1998  and  1999 


The  second  item  focused  more  on  technology  use  in  schools  than  on  classrooms.  As 
shown  in  Figure  5.6,  teachers  in  NAS  and  non-NAS  schools  reported  a  decline  in  technology 
being  used  to  manage  curriculum,  instruction,  and  student  progress.  By  1999, 12  percent  of 
NAS  teachers  agreed  that  such  use  of  technology  described  their  school,  while  21  percent  of 
non-NAS  teachers  agreed. 
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Note;  Percentages  are  based  on  a  total  sample  size  of  40  teachers  -  26  NAS  and  14  Non-NAS. 

Figure  5.6 -Percentage  of  Teachers  who  Reported  that  Technology  was  Definitely  Used  to 
Manage  Curriculum,  Instruction,  and  Student  Progress  in  NAS  and  Non-NAS  Schools, 

Spring  1998  and  1999 


TEACHER-REPORTED  EFFECTS  OF  REFORM 

oin"  surveys  we  asked  NAS  and  non-NAS  teachers  to  report  about  the  whether 
various  facets  of  their  school  improved  over  the  past  three  years  and  whether  the  NAS  design 
being  implemented  had  positive  effects  on  tiieir  professional  work  lives  and  on  their  students. 

Teacher  Reports  of  General  School  Improvement  Over  Past  Three  Years 

For  teacher-reported  outcomes  about  school  unprovement,  we  constructed  a  comjxjsite 
combining  questions  that  asked  teachers  to  report  the  extent  to  which  the  following  had 
changed  in  the  past  3  years,  using  a  5-point  scale  (greatly  worsened  to  greatly  improved):^^ 

•  Student  behavior  (order  and  discipline); 

•  The  school's  relationship  with  parents; 

•  How  the  school  relates  to  the  community; 

•  How  students  get  along  with  other  students;  and 

•  How  teachers  get  along  with  students. 


21 

The  alpha  reliability  for  the  general  school  improvement  composite  was  .89  for  1998  and  .83  for  1999. 
The  range  of  correlations  for  the  individual  items  was  .44  to  .57  in  1998  and  .25  to  .31  in  1999. 
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In  both  1998  and  1999, 56  percent  of  the  teachers  in  tire  NAS  schools  reported  that  the  general 
outcomes  somewhat  or  greatly  improved  (see  Figure  5.7).  Teachers  in  the  non-NAS  schools 
perceived  a  marked  improvement  in  these  factors  on  average  between  1998  and  1999;  in  1998, 
36  percent  of  the  non-NAS  teachers  reported  that  outcomes  improved,  and  in  1999, 50  percent 
reported  such  improvement. 


■  NAS 
□  Non-NAS 


Note;  Percentages  are  based  on  a  total  sample  size  of  40  teachers— 26  NAS  and  14  Non-NAS. 

Figure  5.7— Percentage  of  Teachers  Who  Reported  that  Their  School  Generally  Improved  to  a 
Moderate  or  Great  Extent  Over  the  Past  Three  Years, 

NAS  and  Non-NAS  Schools,  Spring  1998  and  1999 


Teacher  Reports  of  NAS  Design  Effects  on  Teaching  and  Learning 

The  school  improvement  composite  was  comprised  of  several  items  asking  about  the 
extent  to  which  the  NAS  design  had  positive  or  negative  effects  on  individual  teacher' s:  ^ 

•  Teaching; 

•  Professional  growth; 

•  Job  satisfaction; 

•  Students'  achievement; 

•  Students'  enthusiasm  for  learning 

22  The  teacher-reported  effects  composite  was  comprised  of  items  on  a  7-point  scale  (great  deal  of 
negative  effect  to  great  deal  of  positive  effect).  The  alpha  reliability  for  this  composite  was  .93  in  1998  and 
.96  in  1999.  The  range  of  correlations  for  the  individual  items  was  .46  to  .85  in  1998  and  .58  to  .86  in  1999. 
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•  Qassroom  curriculum;  and 

•  Students'  engagement  in  learning. 

In  both  survey  years,  a  substantial  proportion  of  teachers  in  the  NAS  schools  (83  percent 
in  1998  and  88  percent  in  1999)  attributed  positive  effects  of  the  designs  on  these  aspects  of  their 
work  lives  and  on  their  students. 

Few  teachers  felt  that  their  style  of  teaching  had  dramatically  shifted  as  a  result  of  design 
training.  Differences  in  the  way  teachers  approached  curriculum,  however,  was  cited  as  a 
positive  consequence  of  design  implementation.  Several  teachers  reported  that  they  now 
engaged  in  much  more  research  to  prepare  lessons  and  plan  activities.  The  designs  forced  them 
to  study  topics  and  gain  new  content  knowledge,  discouraging  complacency  and  a  reliance  on 
worksheets  and  textbooks. 

At  one  of  our  Co-NECT  schools,  a  teacher  reported  tiiat  because  of  Eie  design,  she 
became  more  aware  that  she  needed  to  involve  the  community  in  her  students'  education  and 
thus  made  a  greater  effort  to  get  parents  or  "some  kind  of  outside  influence"  into  her 
classroom.  Another  teacher  at  a  different  Co-NECT  school  stated  that  the  design  led  him  to 
think  more  deeply  about  subject  matter: 

"When  we  plan,  when  we  look  at  things,  you  look  at  them.. .much  broader,  more  in- 
depth.  And  you're  just  not  dealing  with  surface  things  because  you  really  want  to  get 
involved  in  Ae  product  in  what  you're  doing.  Also,  you  find  yourself  pulling  and 
researching  more  as  a  teacher,  trying  to  find  out  more...about  information,  using  the 
internet  to  get  a  website  and  things  like  books  and  things  like  that... And  I  think  it's 
expanded  what  we  teach." 

At  this  school,  teachers  also  cited  increased  technology  use  as  a  positive  consequence  of 
design  implementation  and  training.  Teachers  were  taught  how  to  work  computers  and  use 
them  in  their  classrooms.  According  to  those  we  spoke  with,  before  Co-NECT,  computers  just 
sat  there  aU  day,  "not  being  used." 

Teachers  at  our  MRSH  schools  remarked: 

"I  like  the  standards  because  they  focus  me.  They  show  me  exactly  what  I'm  doing...We 
used  to  just  come  in  and  teach  a  xmit  and  hope  for  the  best,  you  know... And  now  it's 
much  more  focused...Teaching  is  more  focused.  The  principal  knows  what  we're  doing. 
Everybody  knows  what  we're  doing,  if  we  have  all  the  same  focus." 
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"The  greatest  advantage  to  the  school  the  design  brings  is  that  a  lot  of  teachers  who  sit 
back  and  give  worksheets  can  no  longer  do  that,  and  I  think  that  worksheets  are  taboo 
around  here  now.  So  I  think  it' s  made  a  lot  of  lazy  teachers  less  lazy." 

According  to  a  teacher  at  an  BLOB  school,  much  of  the  training  she  received  "raised  a 
consciousness  or  awareness  in  (her)  to  better  allow  (her)  to  teach  to  the  whole  child,  instead  of 
just  to  academic  flaws."  Teachers  at  our  other  ELOB  school  reported  that  as  a  result  of  having 
the  design  in  place,  they  noticed  some  improvement  in  their  students'  oral  skills.  Additionally, 
the  design  led  them  to  engage  their  students  in  more  group  work  and  research. 

Teachers  at  both  SFA/RW  schools  we  visited  reported  that  they  foimd  it  really 
advantageous  to  work  with  children  at  the  same  reading  level.  SFA  enabled  teachers  to 
"...focus  on  where  the  kids  were  actually  reading  and  take  them  from  there  and  fill  in  some  of 
fire  gaps."  At  one  of  our  SFA/RW  schools,  teachers  stated  that  their  students  were  reading 
more  on  their  own  as  a  result  of  tihe  reading  program.  A  teacher  at  the  other  SFA/RW  school  in 
our  sample  reported  that  several  of  her  colleagues  utilized  the  SFA  reading  strategies  when 
teaching  subjects  that  involved  reading. 

Overall,  teacher  reported  effects  of  reform  in  NAS  schools  revealed  that  over  half  of  the 
NAS  teachers  reported  their  school  had  generally  improved  over  the  last  three  years,  but  by 
1999  about  half  of  the  non-NAS  teachers  also  reported  such  general  improvements.  When 
considering  effects  of  NAS  designs  on  teachers'  professional  life  (teaching  and  professional 
growth)  and  student  learning  and  engagement,  NAS  teachers  tended  to  be  quite  positive.  The 
mixed  picture  may  be  due  to  NAS  teachers  teaching  in  challenged  schools  with  low-achieving 
students  and  having  high  expectations  for  student  progress  in  the  skill  areas. 

OVERARCHING  THEMES 

Analysis  of  responses  to  fire  survey  revealed  few  differences  in  teacher  perceptions  of 
instructional  environments  between  NAS  and  non-NAS  schools.  Some  changes  were  evident. 
For  example,  teachers  in  NAS  schools  reported  instructional  strategies  and  classroom  practices 
that  could  be  categorized  as  reform-like,  rather  than  conventional.  In  other  areas,  fewer 
differences  were  found.  Teachers  in  non-NAS  schools  indicated  more  frequent  use  of 
traditional  assessment  instruments  with  their  students.  Both  groups  reported  similar  use  of 
instructional  materials,  though  more  teachers  in  NAS  than  non-NAS  schools  perceived 
inadequate  materials  to  be  a  problem.  Use  of  technology  in  the  classroom  was  more  prevalent 
in  non-NAS  schools,  but  schoolwide  use  of  technology  was  more  prevalent  in  NAS  schools. 
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The  more  substantial  differences  shown  here,  however,  were  not  between  NAS  and  non- 
NAS  schools,  but  between  1998  and  1999,  which  is  likely  a  reflection  of  the  dramatic  level  of 
change  within  the  district  itself. 

As  shown  in  the  last  chapter,  by  the  end  of  1999,  NAS  teachers  reported  a  substantial  shift  in 
their  perceptions  indicating  little  support  and  stronger  opposition  to  NAS  design  team 
programs  when  compared  with  their  perceptions  a  year  earlier.  Yet,  this  same  group  of 
teachers  expressed  relatively  more  positive  views  about  the  impact  of  NAS  in  both  general  and 
design-specific  terms.  These  contradictory  results  could  be  attributed  to  changes  in  district 
administration  and  the  emphasis  on  accountability  and  TAAS.  Data  collected  in  teacher  focus 
groups,  interviews  and  classroom  observation  give  more  detail  on  the  process  of  change  in  the 
district  and  teacher  attitudes  toward  NAS  and  provide  explanation  and  a  context  for  the  survey 
results  described  here.  Perhaps  the  most  influential  factor  that  posed  a  barrier  to  sustained, 
meaningful  NAS  design  implementation  was  the  high-stakes  assessment  to  which  schools  were 
held  accountable.  How  the  designs  fit  this  accoimtability  system  was  open  to  question.  Yet, 
district  personnel  and  teachers  seemed  to  have  little  time  to  let  this  question  be  resolved.  The 
result  was  more  district  initiatives  aimed  at  changing  teaching  and  learning  that  would  be 
reflected  by  TAAS  scores.  The  NAS  designs  were  but  a  part  of  this  mix  of  educational  reforms. 

District  initiatives  compromised  the  essence  of  designs  by  forcing  the  reduction  of  each  to 
blocks  of  time.  Teachers  reported  "doing  Co-NECT"  and  "getting  to  ELOB  and  MRSH"  upon 
completing  such  activities  as  daily  oral  language  drill,  journal  writing,  reading,  and  math.  To 
be  able  to  meet  the  demands  of  both  district  and  design,  teachers  had  little  alternative  but  to 
turn  their  respective  designs  "on  and  off,"  depending  on  task.  Often,  this  meant  that  teachers 
taught  the  design  almost  as  a  separate  subject  after  completing  reading,  writing  and  math 
lessons.  In  practice,  the  designs  came  to  mean  project  time  to  students.  The  emphases  on 
standards  made  this  less  the  case  at  MRSH  schools.  In  the  second  year  of  the  study,  when  the 
district  prescribed  more  instructional  strategies,  tixe  new  strategies  took  time  away  from 
engaging  in  design-related  interdisciplinary  projects. 

The  teachers  did  not  feel  that  their  designs  necessarily  competed  with  district  initiatives, 
rather  ideas  regarding  which  to  prioritize  were  what  clashed.  They  found  it  difficult  to  actually 
integrate  the  ideas  coming  from  both  sources.  This  struggle  was  made  manifest  as  one  sat  in 
classrooms  and  observed  instruction.  Not  only  was  it  clear  that  teachers  turned  designs  on  and 
off,  most  striking  was  that,  by  the  second  year  of  our  study,  regardless  of  whether  schools  were 
non-NAS  or  had  in  place  Co-NECT,  ELOB,  MRSH,  SFA/RW,  one  saw  similar  instruction  taking 
place. 

Almost  every  teacher  began  his/her  day  with  a  daily  oral  language  drill.  All  teachers 
taught  the  reading  process  and  had  students  write  in  their  "read  and  respond  journals."  AU 
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teachers  employed  Everyday  Math.  All  students  were  taught  spelling  out  of  the  same  district- 
issued  spelling  books.  All  teachers  reviewed  specific  TAAS  skills  and  referred  by  name  to  key 
TAAS  objectives.  All  four  A  graders  were  taught  four  types  of  writing  via  the  writing  process. 

Because  teachers  referred  to  state  standards  when  developing  design  units,  one  saw 
students  across  design  schools  engage  in  similar  activities  and  study  identical  topics.  In  Co- 
NECr,  ELOB,  and  MRSH  schools,  various  instructional  activities  centered  about  Texan  Native 
Americans,  for  example,  were  observed  during  the  1998-1999  school  year. 

Interestingly,  though  introduced  as  design-inspired  projects,  several  identical  activities 
were  performed  across  different  design  schools.  In  Co-NECT  and  ELOB  schools,  for  example, 
the  students  wrote  stories,  using  pictographs,  on  wrinkled  brown  paper  shaped  to  resemble 
animal  skins.  In  ELOB  and  MRSH  schools,  students  constructed  a  variety  of  three-dimensional 
dwellings  inhabited  by  native  Texans. 

In  MRSH  schools,  teachers  got  together  to  develop  instructional  plans  guided  by  the 
state,  drawing  from  design  and  state  standards  as  well  as  Core  Knowledge.  In  ELOB  schools, 
too,  the  teachers  developed  expeditions  around  district  mandates  and  Texas'  education 
standards.  Additionally,  they  derived  inspiration  from  their  ten  design  team  principals, 
incorporating  as  many  as  they  could  into  their  lessons.  Expeditions  were  planned  to  promote 
active  student  involvement  and  to  nurture  oral  language  skills.  Co-NECT  teachers,  like  the 
others  also  developed  curriculxim  based  on  state  standards.  The  outcome  differed  only  in  that 
their  lessons  tended  to  make  more  use  of  technology.  However,  this  observation,  too,  varied  by 
school.  One  Co-NECT  school  more  so  than  another  in  San  Antonio  tended  to  utilize  computers 
to  engage  in  research,  produce  hjrper-studio  reports,  and  type  up  written  work.  Students 
scanned  photographs  into  their  computers  and  tried  to  utilize  the  technological  equipment  they 
had  available  (e.g.  digital  camera  and  camcorder).  However,  even  at  this  school,  plans  often 
derailed  due  to  equipment  failure. 

The  implementation  of  SFA/RW  in  San  Antonio  differed  from  the  other  three  designs  in 
that  SFA/RW  schools  opted  to  implement  only  one  aspect  of  the  design:  Success  for  All  (SFA). 
This  decision  tended  to  ease  the  pressure  that  teachers  at  other  design  schools  felt  to  "fit" 
design  elements  into  their  days.  Given  that  SFA  is  a  highly  structured,  prescribed  reading 
program,  an  academic  area  of  critical  importance  to  the  district,  teachers  in  SFA/RW  schools 
expressed  greater  confidence  in  their  progress  and  degree  of  implementation  than  tiieir 
colleagues  at  other  design  schools.  By  electing  to  adopt  only  SFA,  implementation  was  made 
less  challenging  given  its  limited  nature. 

Clearly,  district-issued  curricular  and  instructional  strategies  regarding  reading,  language 
arts,  and  math  limited  the  ability  of  teachers  to  develop  units,  expeditions,  and  lessons  as 
described  in  their  respective  design  literature.  Moreover,  because  the  schools 
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were  obligated  to  use  the  same  math  curriculum  and  engage  in  90  minutes  of  uninterrupted 
math  and  reading  instruction  and  70  or  so  minutes  of  language  arts  activity,  teachers  tended  to 
develop  design  units  around  social  studies/ science  topics.  Design  activities  were  frequently 
scheduled  for  the  afternoon. 

Due  to  time  constraints  during  tire  school  day,  teachers  across  the  various  design  schools 
also  revealed  that  they  tended  not  to  complete  their  units  or  expeditions  as  planned.  So  many 
unaccounted  factors  interrupted  the  flow  of  their  ^^units^^  diat  time  was  lost.  Because  time  was 
of  the  essence,  teachers  tended  to  move  on  to  different  units  without  producing  final 
culminating  products. 

Instead  of  NAS  designs  guiding  curriculum  and  instruction,  it  appeiirs  that  the  district 
and  state  initiatives  directed  the  educational  mission  of  all  schools.  TAAS  success  was 
obviously  the  major  driving  force  behind  all  this.  The  district's  influence  was  clearly  revealed 
in  the  conversations  we  had  with  teachers,  which  explain  many  of  our  survey  results  showing 
few  differences  between  NAS  and  non-NAS  teachers.  Regardless  of  design  adoptions,  teachers 
across  the  various  San  Antonio  schools,  tended  to  voice  the  same  frustrations.  As  stated, 
limited  school  hours  and  demands  on  teacher  time  were  repeatedly  discussed  at  all  schools. 
Teachers  frequently  mentioned  their  difficulty  trying  to  "fit"  all  district  and  design  activities 
into  any  one  school  day.  Teachers  remarked  that  the  emphasis  on  TAAS  tended  to  stifle  their 
creativity.  They  were  given  little  opportunity  to  devise  their  own  ways  to  meet  the  needs  of 
their  students. 

All  teachers  expressed  their  annoyance  at  having  to  devote  energy  and  time  to  seemingly 
petty  activities,  for  example,  checking  students'  math  and  writing  journals  for  dates  and  times. 
They  expressed  that  full-blown  design  implementation  was  difficult  to  accomplish  given  ffie 
variety  and  frequency  of  interruptions  experienced  throughout  the  year.  At  the  beginning  of 
the  1998-1999  school  year,  for  example,  teachers  across  schools  were  asked  to  attend  district- 
sponsored  workshops  and/ or  training  during  the  school  day  at  least  once  a  week.  This  was  a 
source  of  much  trouble.  Being  away  tended  to  mean  that  students  feU  behind.  Other 
interruptions  reported  by  teachers  included:  TAAS  simulations;  preparation  for  TAAS; 
meetings  called  after  school  (often  imexpectedly)  to  discuss  district  initiatives,  taking  time 
away  from  teachers'  to  organize  their  classrooms  and  plan  lessons;  the  introduction  of  district 
initiatives  during  teacher  planning  periods;  the  strict  math  pacing  guide;  the  required 
administration  of  periodic  math  tests;  and  the  paperwork  required  to  document  classroom 
activities. 


"...the  district' s  programs  come  before  design.  That's  tire  way  it  is." 
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" We've  got  to  get  the  kids  prepared  for  (TAAS).  So  we're  taking  the  time  out  to  do  that, 
to  get  the  kids  ready  for  it  and  we're  operating  in  two  different  mindsets..." 

"...I  don't  know  where  our  direction  is  at.  I  know  that  the  district  is  becoming  more  and 
more  a  top-down  situation  where  we're  told  this  is  what  the  reading  is  going  to  look 
like.  When  district  staff  walk  in,  they  want  to  see  the  reading  done  in  a  certain  way." 

Our  interviews  revealed  that  though  the  district  was  supportive  both  financially  and 
philosophically  of  NAS  designs  in  its  schools,  it  unwittingly  hindered  design  implementation 
at  all  schools  except  SFA/RW  by  establishing  an  ever-growing  presence  in  the  daily  classroom 
affairs  of  its  teachers.  The  paucity  of  commxmication  between  the  district  and  design  teams 
failed  to  create  the  kind  of  supportive  operating  environment  called  for  by  NAS.  Moreover,  the 
limited  communication  between  teachers  and  their  respective  design  representatives  served  to 
weaken  implementation  as  well.  Not  knowing  how  to  integrate  central  office  initiatives  with 
design  aspects,  teachers  tended  to  compromise  designs  by  selecting  and  modifying  those 
elements  only  that  could  coexist  with  district  actions. 

"I  just  think  that  (the  district)  is  trying  to  do  too  many  things.  Maybe  they  feel  that  our 
schools  are  very  low  so  they  are  doing  all  these  other  things  without  really  giving  us  a 
chance  to  test  it.. .They  are  doing  all  these  things  without  realizing  that  it's  overkill.  It's 
way  too  much." 

Despite  the  similarities  across  the  district  in  instructional  conditions  and  the  press  of 
district  reading  and  mathematics  initiatives  and  the  marginalization  of  NAS  designs,  it  is  worth 
exploring  the  relationships  between  instructional  conditions  that  were  mostly  similar  across  the 
district  on  student  achievement  on  the  TAAS  reading  and  mathematics  tests  as  well  an 
independent  reading  commercial  assessment  (Stanford-9).  It  is  to  this  that  we  turn  in  the  next 
chapter,  first  examining  relationships  in  all  the  4*  grade  classrooms  and  students,  and  second 
in  the  sample  of  classrooms  and  students  for  which  RAND  gaffiered  additional  survey 
information  on  classroom  conditions  and  supplemental  achievement  scores. 
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6.  EFFECTS  OF  INSTRUCTIONAL  CONDITIONS  ON  STUDENT  ACHIEVEMENT 


The  ultimate  aim  of  school  reform  efforts  and  implementation  of  NAS  designs  is  to 
substantially  improve  student  performance.  In  this  chapter  we  turn  to  a  quantitative  analysis 
of  test  scores  and  examine  the  relative  impact  of  factors  influencing  student  performance.  The 
empirical  models  constructed  allow  for  an  exploratory  analysis  of  classroom  effects  of  reform  in 
a  low-performing  school  district.  Moreover,  at  tfie  height  of  design  implementation  in  San 
Antonio,  curricular  differences  between  NAS  and  non-NAS  schools  were  evident.  Here  we 
explore  the  possibility  of  differences  in  student  achievement  through  a  comparison  of 
conditions  within  NAS  and  non-NAS  classrooms,  taking  into  account  the  nesting  of  students 
within  classrooms  within  schools.  What  follows  is  not  a  test  of  NAS  design  effects,  per  se, 
because  schools  and  classroom  were  at  the  early  stages  in  implementation.  Rather,  we  focus  on 
whether  differences  in  instructional  conditions  are  related  to  student  achievement,  net  of  other 
student,  classroom,  and  school  factors.^s  Particularly,  we  are  interested  in  whether  reform-Uke 
instructional  conditions  are  related  to  student  achievement  because  these  are  the  practices  that 
are  consistent  with  what  NAS  design  teams  are  attempting  to  promote. 

Given  the  available  data,  we  conducted  two  sets  of  analyses:  First,  for  the  entire  district 
we  examined  the  effects  of  student,  teacher,  and  school  characteristics  on  the  4*  grade  TAAS 
reading  and  mathematics  scores.  Data  provided  by  the  San  Antonio  district  and  other  sources 
allowed  for  construction  of  a  data  set  containing  more  than  3,800  fourth  grade  students  in  about 
280  classrooms  in  all  64  elementary  schools  in  the  district.  Individual  TAAS  reading  and 
mathematics  scores  were  regressed  against  student,  teacher,  classroom,  and  school 
characteristics  using  multilevel  linear  models  to  partition  the  variation  in  reading  and 
mathematics  achievement  into  student  and  classroom  components.  Second,  we  analyzed 
student  achievement  in  a  sub-sample  of  over  800  students  in  63  classrooms  for  which  teachers 
completed  our  survey. 

The  results  at  the  district  level  provide  the  context  for  the  sub-sample.  Data  gathered 
from  the  teacher  surveys  help  inform  the  district  analysis  on  the  impacts  of  teacher  practices 
and  perceptions  of  student  achievement.  In  addition,  these  students  were  administered  the 
Stanford-9  open-ended  reading  test,  making  possible  an  independent  measure  of  student 
performance  without  the  "high  stakes"  implications  of  the  TAAS. 

23  It  is  important  to  note  tiiat  while  we  examine  matihematics  achievement,  the  specific  curriculum 
(Everyday  Mathematics)  being  implemented  was  likely  the  primary  driver  of  test  scores,  not  the  other 
reforms  being  implemented.  Yet,  it  is  informative  to  examine  subjects  other  than  reading  to  explore  the 
effects  of  general  instructional  conditions  on  student  achievement. 
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In  the  sections  that  follow,  we  discuss  the  operationalization  of  the  variables  included, 
identify  the  models  estimated,  describe  the  methods  used,  and  analyze  the  results  of  those 
regressions. 

DISTRICT  LEVEL  DATA  AND  DEPENDENT  VARIABLES 

In  each  of  the  models  presented,  individual  test  scores  are  used  to  measure  student 
achievement.  The  district-wide  data  set  consists  of  all  fourth  grade  students  in  San  Antonio 
during  the  1997-1998  school  year  who  had  valid  scores  on  both  the  reading  and  mathematics 
sections  of  the  TAAS.^^ 

Because  of  the  sweeping  changes  in  the  district  that  year  and  key  differences  between 
data  sets  provided,  we  decided  to  limit  the  analysis  to  the  1997-1998  school  year,  even  though 
data  from  the  1998-1999  school  year  were  also  available.  During  site  visits  and  classroom 
observations  in  spring  1998,  design  implementation  was  clearly  underway  in  many  schools.  By 
the  same  time  in  1999,  however,  implementation  of  NAS  designs  had  been  pushed  to  the 
backgroxmd.  The  district  had  mandated  more  tmiform  reforms.  As  described  in  the  previous 
chapters,  instruction  in  all  classrooms  was  fairly  generic  and  focused  on  preparation  for  the 
TAAS.  Any  effects  from  NAS  implementation  would  be  more  likely  found  in  the  1997-1998 
data  set. 

At  the  district  level,  Texas  Learning  Index  (TLI)  scores  from  the  reading  and  mathematics 
sections  of  the  TAAS  are  the  dependent  variables  in  separate  sets  of  regression  models.25  For 
ease  of  interpretation,  these  scores  are  standardized  to  have  a  mean  of  zero  and  a  standard 
deviation  of  one.^^ 

24  The  original  data  files  provided  by  San  Antonio  contained  records  on  4,509  students.  Of  these,  many 
did  not  have  test  scores  because  they  were  enrolled  in  alternative  education  centers  or  were  exempt  from 
the  TAAS  because  of  special  education  or  limited  English  proficiency  status.  A  smaller  proportion  of 
students  were  excluded  from  the  sample  because  one  or  both  of  their  scores  were  missing.  Of  these,  some 
may  have  been  absent  when  the  TAAS  was  administered  or  had  invalid  results  because  of  coding 
irregularities.  On  average,  we  found  no  statistically  significant  differences  in  characteristics  between 
students  who  were  not  exempt  from  the  TAAS,  but  did  not  have  valid  scores,  and  those  included  in  the 
data  set  used  for  the  models  we  analyze  here. 

2^  These  scores  are  transformations  of  raw  estimates  of  student  ability  scaled  to  have  a  mean  of  70 
(reflecting  the  standard  for  passing)  and  a  standard  deviation  of  15.  Succinctly,  "the  TLI  is  not  a 
percentage  of  items  correct.  It  is  a  standard  score.  Its  primary  functions  are  to  describe  how  far  above  or 
below  the  passing  standard  flie  student  is  and  to  indicate  whether  the  student  is  making  learning  progress 
over  time" (Texas  Student  Assessment  Program  Technical  Dieest  for  the  Academic  Year  1999-2000,  page 
30;  Texas  Education  Agency; 

http://www.tea.state.tx.us/student.assessment/resources/techdig/chap5.pdf). 

26  The  TLI  scores  are  significantly  skewed  to  the  right  To  satisfy  normality  assumptions  in  our  estimation 
methods,  before  standardizing  the  TLI  scores,  we  transformed  each  set  of  scores  using  the  Box-Cox 
method  (for  details  see  Greene  (2000:  p.  444). 
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OFERATIONALIZING  THE  INDEPENDENT  VARIABLES 
Student  Characteristics 

Students  in  SAISD  come  from  fairly  homogeneous  backgrounds.  Most  are  Latino,  and 
from  low-income  families.  Nevertheless,  it  is  important  to  rmderstand  how  differences 
between  individual  students,  such  as  their  socio-economic  characteristics,  English  language 
ability,  mobility  patterns,  and  prior  achievement  affect  their  performance  as  measured  by 
standardized  test  scores. 

In  our  models,  we  include  dummy  variables  for  each  student' s  gender  (male  as 
reference),  and  race  or  ethnicity  (African  American,  white  or  other,27  vs.  Latino).  Many 
students  in  Texas  whose  primary  language  is  Spanish  are  exempt  from  the  TAAS  and  instead 
take  a  version  of  the  instrument  in  Spanish.  Olliers,  however,  are  not  considered  exempt,  even 
though  they  are  enrolled  in  bilingual  education  programs  or  classified  as  limited  English 
proficient  (LEP).  We  constructed  dummy  variables  to  accoimt  for  biUngual  or  LEP  students  in 
the  data  set. 

Eligibility  for  free  or  reduced  lunch  programs  is  used  as  a  proxy  for  low  income.  San 
Antonio  is  a  Title  I  district,  so  all  students  are  automatically  enrolled  in  free  limch  programs. 
The  district,  however,  keeps  track  of  family  income  and  calculates  which  students  would 
normally  be  eligible  for  these  programs.  A  dummy  variable  was  constructed  using  this 
information  provided  by  the  district.  We  also  include  dummy  variables  for  students  in  special 
education  and  gifted  and  talented  programs.  Age  is  a  continuous  variable,  calculated  from 
each  student' s  date  of  birtii  up  to  the  last  day  of  April,  1998,  the  month  when  the  TAAS  was 
administered. 

We  include  a  measure  of  student  mobility,  calculated  from  dates  of  entry  and  withdrawal 
provided  by  the  district  and  defined  as  the  total  number  of  weeks  student  spent  in  the 
classroom  where  the  TAAS  was  administered. 

As  a  proxy  for  prior  student  achievement,  we  include  the  1997  TAAS  reading  or 
mathematics  TU  scores  corresponding  to  the  dependent  variable.  These  test  scores  from  the 
previous  school  year  are  also  standardized  to  have  a  mean  of  zero  and  standard  deviation  of 
one.  For  those  students  who  did  not  have  valid  scores  from  1997  we  impute  the  mean  value 
and  create  a  missing  score  indicator  dummy  variable  to  control  for  ihis  imputation. 

The  number  of  Asian  American/Pacific  Islander,  and  Native  American  students  is  very  small  in  this 
sample  and  flieir  test  scores  do  not  differ  significantly  from  the  scores  of  White  students,  so  these  four 
categories  were  combined  into  "white  or  other." 
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Teacher  Background  and  Classroom  Characteristics 

As  discussed  in  Chapter  One,  understanding  the  relationship  between  student 
achievement  and  classroom  environments  is  key  to  this  analysis.  Furthermore,  the  particular 
composition  of  students  within  classrooms  may  also  be  an  important  determinant  of  student 
ability  and  we  examine  the  relationship  between  average  classroom  characteristics  and 
achievement. 

We  include  teacher  background  characteristics,  such  as  gender,  race  or  ethnicity, 
educational  degree,  and  years  of  teaching  experience.  Operationalization  of  these  measures  is 
straightforward.  Dummy  variables  were  constructed  for  gender  (female  as  reference),  race  or 
ethnicity  (African  American,  white  or  other  vs.  Latino),  and  educational  degree  (master's  vs. 
bachelor's).  Also  included  are  the  total  years  teaching  experience. 

Measures  designed  to  control  for  classroom  effects  were  constructed  by  taking  each  of  the 
individual  student  characteristics  and  calculating  means  for  aU  the  students  linked  to  a 
particular  teacher.  The  first  set  of  variables  at  this  level  includes  the  proportion  of  students  in 
each  classroom  who  are  male,  African  American,  white  or  other,  enrolled  in  a  bilingual 
education  program,  classified  as  LEP,  eligible  for  free  or  reduced  lunch,  in  special  education  or 
in  gifted  and  talented  programs.  The  other  variables  at  the  classroom  level  consist  of  means 
calculated  for  student  age,  number  of  weeks  in  the  classroom,  and  the  appropriate  1997  TAAS 
reading  or  mathematics  score  (with  respect  to  the  dependent  variable). 

School  Characteristics 

Within  our  two-level  framework,  the  proportion  of  variance  that  can  be  directly 
attributable  to  differences  between  schools,  rather  than  between  classrooms,  is  not  specifically 
calculated.  It  is  still  important,  however,  to  take  into  the  account  the  effect  of  school-wide 
characteristics  on  student  performance.  We  include  two  such  variables  here.  The  first  is  a 
proxy  for  relative  school  quality.  Each  year  the  Texas  Education  Agency  issues  Accormtability 
Ratings,  based  primarily  on  TAAS  passage  rates,  and  attendance  rates.^  We  assigned  each  of 
the  four  possible  ratings  a  number  from  one  to  four  (i.e.,  l=Low  performing,  2= Acceptable, 
3=Recognized,  4=Exemplary)  and  for  each  school  in  the  district  sample  we  calculated  average 
TEA  ratings  from  1995-1997  (i.e.,  from  the  inception  of  the  rankings  to  the  year  prior  to  our 
achievement  data). 

A  second  school-wide  variable  captures  differences  between  NAS  and  non-NAS  schools. 
Since  the  impact  of  a  NAS  design  on  a  given  campus  may  develop  over  time,  the  variable  is  the 


^^For  more  information,  see  http:  /  / www.tea.state.tx.us/ perfrer>ort/accotmt/ . 
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number  of  years  a  school  has  been  implementing  the  design.  We  give  non-NAS  schools  a  zero 
to  allow  the  variable  to  fimction  as  a  weighted  dummy  for  NAS  schools.29 

STUDENT  ACHIEVEMENT  IN  SAN  ANTONIO:  MULTILEVEL  ANALYSIS 

Since  students  are  nested  within  classrooms,  which  are  nested  within  schools,  we  relied 
on  multilevel  modeling  techniques  to  provide  more  accurate  estimates  of  student,  classroom, 
and  school  level  effects  (see  Bryk  and  Raudenbush,  1992;  Bryk,  Raudenbush,  and  Congdon, 
1996;  Singer,  1998). 

Using  district-wide  data,  the  models  estimated  for  both  reading  and  mathematics  scores 
are  specified  as  follows: 

Individual  Student  Model  (Level  1) 

^ij  =  Poj  +  Pi  (Male)  +  P2( African  American)  +  P3(White  or  Other)  +  p4(Bilingual 

Education  Program)  +  P5(Limited  English  Proficiency)  +  p6(Free/Reduced  Lunch 
Program  Eligible)  +  P7(Special  Education)  +  p8(Gifted  and  Talented  Education)  + 
p9(Age)  +  Pio(Weeks  with  Class)  +  pii(1997  TAAS  Score)  +  pizfMissing  1997  TAAS 
Score)  +  Tij, 
where 

•  Yy  is  the  dependent  variable,  i.e.,  the  TAAS  reading  or  mathematics  score  for 
student  /in  classroom /; 

•  Poj  is  the  level  1  constant  term  and  the  average  value  of  tfie  dependent  variable  in 
classroom  /; 

•  pi.i2  are  the  level  1  coefficients  for  the  listed  independent  variables; 

•  ry  is  the  level  1  random  effect. 

Classroom  Context  Model  (Level  2) 

Poj  =  7b+  Yi(Male  Teacher)  +  Y2( African  American  Teacher)  +  Y3(White  or  Other  Teacher) 
+  Y4(Teacher' s  Years  of  Experience)  +  Y5(Master's  Degree  or  Higher)+  Y6(Male.y)  + 

Y7( African  American.^)  +  Y8(White  or  Other.^)  +  Y9(Bilingual  Education  Program.^)  + 
Yio(Limited  English  Proficiency.;)  +  Yu(Free/Reduced  Lunch  Program  Eligible.;)  + 
Yi2(Special  Education.;)  +  Yi3(Gifted  and  Talented  Education.;)  +  Yi4(Age.;)  + 
YisfWeeks  with  Class.;)  +  Yi6(1997  TAAS  Score.;)  +  Yi7(Missing  1997  TAAS  Score.;) 

+  Yi8( Average  TEA  Rating,  1995  to  1997)  +  Yi9(Years  Implementing  a  NAS  Design) 

+  u. 


OQ 

Given  the  available  data,  it  was  also  possible  to  estimate  design  specific  models.  The  results  of  these, 
however,  did  not  enhance  the  present  analysis  and  were  not  included  here. 
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where 

•  Poj  in  this  model  is  from  the  student  level  equation  above,  and  it  is  the  average 
test  score  in  classroom  /; 

•  yo  is  the  constant  term; 

•  yi-18  are  the  level  2  coefficients  for  the  listed  independent  variables;  and 

•  w  is  a  level  2  random  effect. 

The  multi-level  models  describe  above  are  simple  ''fixed  coefficient"  models  (Kreft  and 
DeLeeuw,  1998).  That  is,  the  coefficients  for  the  Level  1  relationships  between  student 
characteristics  and  achievement  scores  are  held  constant  across  classrooms.  There  are  no  cross- 
level  interactions  between  classroom  and  student  characteristics.  Thus,  between-classroom 
differences  are  limited  to  differences  in  intercepts.  In  other  words,  the  intercept  for  each 
classroom  is  the  sum  of  the  overall  intercept  and  the  sums  of  the  classroom  aggregate  variables 
weighted  by  the  classroom-level  regression  coefficients,  plus  random  error.  The  achievement 
score  of  each  student  then  is  the  sum  of  that  student' s  classroom  intercept  and  the  sum  of  the 
student-level  coefficients,  plus  error  (Koretz,  McCaffrey,  and  Sullivan,  2001). 

In  order  to  facilitate  interpretation  of  the  intercept  term  in  the  models,  we  centered  all 
variables  that  were  unique  to  Level  2  at  their  grand  means.  Centering  is  particularly  important 
for  the  Level  1  variables  both  for  interpretation  of  the  intercept  as  well  as  to  ensure  numerical 
stability  (Bryk  and  Raudenbush,  1992:  25).  Without  centering,  one  would  interpret  the  intercept 
as  the  expected  outcome  for  a  student  in  classroom  j  who  has  a  value  of  zero  on  all  the  Level  1 
predictor  variables,  which  often  does  not  have  meaning.  With  centering,  one  can  interpret  the 
intercept  as  the  expected  outcome  for  a  student  in  classroom  j  who  is  at  the  mean  of  all  the 
predictor  variables.  Dummy  variables  were  also  group-mean  centered;  thus  the  intercept  term 
is  the  adjusted  mean  outcome  in  classroom  j,  adjusted  for  differences  among  units  in  the 
percentages  of  students  with  various  characteristics.  Centering  the  Level  2  variables  simply 
adds  to  the  convenience,  so  that  the  intercept  term  can  be  interpreted  as  the  expected  outcome 
for  a  classroom  that  is  at  the  mean  of  the  sample  in  terms  of  classroom  characteristics. 

The  variables  that  were  included  at  both  levels  were  group-mean  centered  at  the  student 
level  (by  subtracting  the  classroom  mean  score  from  each  individual  student' s  score)  and  the 
classroom  means  were  then  entered  at  Level  2  into  the  model.  As  Bryk  and  Raudenbush  (1992) 
point  out,  this  procedure  avoids  the  assumption  that  the  effects  of  variations  in  classroom 
means  equals  the  effects  of  deviations  within  classrooms  and  makes  the  model's  coefficient 
straightforward  estimates  of  the  within-  and  between-classroom  effects.^^ 

Cohen,  Baldi,  and  Rathbun  (1998)  point  out  the  importance  of  reintroducing  the  classroom  means 
when  the  Level  1  variables  are  group-mean  centered.  They  argue  that  a  model  in  which  ihe  (Level  1) 
variables  are  centered  around  their  classroom  means  sheds  an  important  piece  of  information:  the 
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ANALYSIS  OF  THE  DISTRICT  SAMPLE 

For  reference,  the  means  and  standard  deviations  for  the  variables  in  both  the  TAAS 
reading  and  mathematics  models  are  reported  in  Table  6.1 

Student  CharacteristicsMost  fourtii  graders  in  SAISD,  about  84  percent,  are  Latino. 

The  next  largest  group  is  also  minority  as  11  percent  are  African  American.  White  students 
accormt  for  only  about  5  percent  of  the  district  sample.  Qose  to  90  percent  of  students  are  from 
low  income  families  and  are  eligible  for  free  or  reduced  lunch  prices.  Students  who  had  valid 
scores  on  both  TAAS  exams,  and  were  also  either  enrolled  in  a  bilingual  education  program  or 
classified  as  LEP  made  up  approximately  10  percent  of  the  sample.  Over  11  percent  of  the 
sample  consists  of  special  education  students  who  were  not  exempt  from  taking  the  TAAS. 

Of  particular  note  is  the  high  mobihty  rate  for  the  district  sample.  We  set  the  maximum 
number  of  weeks  of  instruction  as  the  weeks  prior  to  the  TAAS  administration  in  April  38.  The 
average  student  in  the  sample  missed  up  to  two  weeks  of  classroom  time  due  to  transfers 
between  teachers  or  schools.  During  the  1997-1998  school  year,  almost  half  of  students  in  the 
sample  were  in  NAS  schools,  and  almost  of  half  of  these  students  were  in  SFA/RW. 

Teacher  Background  and  aassroom  Characteristichess  than  12  percent  of  fourth 
grade  teachers  in  San  Antonio  are  male,  and  most  are  either  Latino  (about  48  percent)  or  white 
(about  37  percent).  The  average  years  teaching  experience  is  almost  14  years,  with  a  standard 
deviation  of  about  nine  years.  A  little  over  a  third  (about  38  percent)  of  teachers  in  this  larger 
district  sample  have  earned  master's  degrees.  The  percent  of  teachers  in  NAS  schools  and 

ckssroom  mean  of  the  variable.  Specifically,  they  write,  "When  the  analyst  fails  to  re-introduce  this 
source  of  systematic  variation  appropriately  elsewhere  in  the  model,  he  or  she  posits  that  the  actual  value 
of  the  centered  does  not  influence  the  outcome,  only  the  relative  value  (that  is,  relative  to  the  school  [or 
classroom  in  this  context]  mean)"  (1998: 18-19). 
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Table  6.1 

Descriptive  Statistics  of  Variables  in  Multilevel  District  Sample  Analysis 


Variables 

Mean 

SD 

Dependent  Variables 

TAAS  Reading  Score,  1998 

75.03 

16.19 

TAAS  Mathematics  Score,  1998 

71.58 

15.40 

Independent  Variables  for  Students  (n  =  3820) 

Percent  male 

49.06 

Race  or  ethnicity 

Percent  African  American 

10.89 

Percent  Asian  American/Pacific  Islander 

0.10 

Percent  Latino 

83.56 

Percent  Native  American 

0.10 

Percent  White 

5.29 

Percent  in  Bilingual  Education  program 

11.91 

Percent  with  limited  English  proficiency 

5.18 

Percent  Free/ Reduced  Lunch  program  eligible 

88.53 

Percent  in  Special  Education 

11.18 

Percent  in  Gifted  and  Talented  education 

6.91 

Age  (n=3819) 

10.31 

0.45 

Weeks  in  classroom 

35.98 

7.30 

1997  XAAS  reading  score  (n=3,195) 

66.06 

22.78 

1997  TAAS  mathematics  score  (n=3,116) 

69.56 

16.83 

Percent  of  Students  in  NAS  Schools  (n  =  1906) 

49.90 

Percent  in  Co-NECT  Schools  (n  =  229) 

5.99 

Percent  in  ELOB  Schools  (n  =  190) 

4.97 

Percent  in  MRSH  Schools  (n  =  574) 

15.03 

Percent  in  SFA/RW  Schools  (n  =  913) 

23.90 

Independent  Variables  for  Teachers  and  Classrooms  (n  =  279) 

Percent  Female 

11.47 

Total  years  teaching 

13.53 

8.% 

Teacher  race  or  ethnicity 

Percent  African  American 

13.62 

Percent  Asian  American/Pacific  Islander 

0.36 

Percent  Latino 

48.39 

Percent  Native  American 

0.36 

Percent  White 

37.28 

Percent  with  Master's  degree 

38.35 

Classroom  percent  male 

49.15 

0.19 

Classroom  race  or  ethnicity 

Percent  African  American 

10.06 

0.20 

Percent  White  or  other 

4.93 

0.10 

Classroom  percent  bilingual  education  program 

21.35 

0.38 

Classroom  percent  limited  English  proficiency 

5.33 

0.11 

Classroom  percent  free/reduced  lunch  program  eligible 

89.33 

0.13 

Classroom  percent  special  education 

14.91 

0.22 

Classroom  percent  gifted  and  talented  education 

5.8 

0.10 

Classroom  average  age 

10.34 

0.26 
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Table  6»1  (cont.) 


Classroom  average  weeks  in  class 

Classroom  average  1997  TAAS  reading  score 

Classroom  average  1997  TAAS  mathematics  score 

Percent  Teaching  in  NAS  Schools  (n  =  138) 

Percent  Teaching  in  Co-NECT  Schools  (n  =  15) 

Percent  Teaching  in  BLOB  Schools  (n  =  13) 

Percent  Teaching  in  MRSH  Schools  (n  =  40) 

Percent  Teaching  in  SFA/RW  Schools  (n  =  70) 

36.12 

63.55 

68.89 

49.46 

5.38 

4.66 

14.34 

25.09 

2.44 

14.35 

8.58 

Independent  Variables  for  Schools  (n  =  64) 

1995-1997  Average  TEA  Rating 

2.00 

0.21 

Years  Implementing  NAS  Design  (NAS  Schools  only  -  n  =  32) 

1.47 

0.67 

Percent  of  Schools  Implementing  a  NAS  Design  (n  =  32) 

50.00 

Percent  Implementing  Co-NECT  Design  (n  =  3) 

4.69 

Percent  Implementing  ELOB  Design  (n  ==  3) 

4.69 

Percent  Implementing  MRSH  Design  (n  =  9) 

14.06 

Percent  Implementing  SFA/RW  Design  (n  =  17) 

26.56 

linked  to  specific  designs  is  roughly  proportional  to  the  distribution  of  students  between  NAS 
and  non-NAS  schools  and  within  specific  designs. 

Given  the  distribution  of  students,  we  did  not  have  information  on  class  size  and  could 
not  attach  any  kind  of  weighting  for  class  size  within  the  model  specification  to.  The  classroom 
means  calculated  for  each  of  the  student  characteristics  all  roughly  correspond  to  the  individual 
student  means. 

School  Characteristics  The  mean  of  2.0  for  TEA  rankings  from  1995-1997  reflects  the  fact 
that  almost  all  the  schools  in  the  sample  had  Acceptable  ratings  during  these  years.  On  average, 
schools  had  been  implementing  NAS  designs  for  one  or  two  years  prior  to  the  1998  TAAS 
administration. 

MULTIVARIATE  RESULTS  FOR  SAN  ANTONIO 

We  estimated  the  model  above  for  both  the  TAAS  reading  and  mathematics  scores  to 
examine  the  relationships  between  student  achievement  and  student  and  classroom 
characteristics.  The  estimation  results  are  given  in  Table  6.2.  Since  the  dependent  variables 
have  been  standardized,  a  coefficient  represents  the  increment  change  in  the  dependent  from  a 
unit  increase  in  the  independent  variable. 
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Table  6.2 

Multilevel  Results  for  the  Relationships  of  District-wide  1998  4th  Grade  TAAS  Scores  to 

Student,  Qassroom,  and  School  Factors 


Reading _ Mathematics 


Variables 

Coefficient 

SE 

Coefficient 

SE 

Intercept 

0.31 

1,34 

2,75 

1.48 

Independent  Variables  for  Students  (n  =  3820) 

Male 

-0.03 

0.02 

0.02 

0.02 

African  American 

-0.08 

0.05 

-0.01 

0.04 

White  or  other 

0.15" 

0.05 

0.11'’ 

0.05 

Biiingiial  education  program 

0.00 

0.08 

-0.03 

0.08 

Limited  English  proficiency 

0,05 

0.06 

0.00 

0.06 

Free/reduced  lunch  program  eligible 

-0.14^ 

0.05 

-0.09> 

0.04 

Special  education 

-0.47^ 

0.04 

-0.51" 

0.04 

Gifted  and  talented  education 

0.51" 

0.05 

0.39" 

0.05 

Age 

-0.09^ 

0.03 

-0.10" 

0.02 

Weeks  in  classroom 

0.01" 

0.00 

0.00# 

0.002 

1997  TAAS  reading  score 

0.53" 

0.01 

c 

c 

1997  TAAS  mathematics  score 

c 

c 

0.57" 

0.01 

Missing  1997  TAAS  reading  score 

-0,11" 

0.04 

c 

c 

Missing  1997  TAAS  mathematics  score 

c 

c 

-0.21" 

0.04 

Independent  Variables  for  Classrooms  and  Teachers  (n 

"=279) 

Male  Teacher 

-0.02 

0.06 

-0.05 

0.07 

Years  of  Teaching  Experience 

-0.05 

0.06 

-0.04 

0.07 

African  American  Teacher 

-0.08 

0.05 

-0.05 

0.05 

White  or  Other  Teacher 

0.00 

0.00 

0.00 

0.00 

Master’s  Degree 

-0.02 

0.04 

-0.03 

0.05 

Classroom  percent  male 

-0.31?> 

0.14 

-0.16 

0.15 

Classroom  percent  African  American 

0.09 

0.10 

-0.09 

0.12 

Classroom  percent  White  or  Other 

0.61^ 

0.25 

0.69> 

0.29 

Classroom  percent  Bilingual  Education  Program 

-0.11 

0.08 

-0.05 

0.08 

Classroom  percent  Limited  English  Proficiency 

0.02 

0.20 

-0.18 

0.23 

Classroom  percent  Free/Reduced  Limch  Program  Eligible 

-0.07 

0.21 

-0.07 

0.25 

Classroom  percent  Special  Education 

-0.41?’ 

0.16 

-0.76" 

0.16 

Classroom  percent  Gifted  and  Talented  Education 

0.91" 

0.21 

0.80" 

0.25 

Classroom  average  Age 

-0.12 

0.12 

-0.3# 

0.13 

Classroom  average  Weeks  in  Classroom 

0.01 

0.01 

0.01 

0.01 

Classroom  average  1997  TAAS  Reading  Score 

0.54" 

0.06 

c 

c 

Classroom  average  1997  TAAS  Mathematics  Score 

c 

c 

0.45" 

0.06 

Classroom  percent  Missing  1997  TAAS  Reading  Score 

-0.13 

0.14 

c 

c 

Classroom  percent  Missing  1997  TAAS  Mathematics  Score 

c 

c 

-0.18 

0.15 
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_ _ _ Table  6.2  (cont.) _ 

Independent  Variables  for  Schools  (n  =  64) 

Average  Texas  Education  Agency  Rank,  1995-1997  0.35*  0.11  0.26’  0.13 

Years  Implementing  a  NAS  Design _  -0.03 _ 002 _ -0.03  0.03 

NOTES:  “Significant  at  0.01  level  (highlighted  in  bold) 

‘’Significant  at  0.05  level  (highlighted  in  bold) 

“^Excluded  from  the  model 


STUDENT  LEVEL  EFFECTS 

In  general,  the  socioeconomic  factors  examined,  (race  or  ethnicity  and  family  income), 
were  important  predictors  in  the  models.  Students  in  the  white/other  category  tend  to  perform 
better  than  Latinos  by  .15  of  a  standard  deviation  (SD)  in  reading  and  .11  of  a  SD  in 
mathematics.  Students  who  were  eligible  for  free  or  reduced  lunch  programs  had  significantly 
lower  scores  than  students  who  were  not  eligible  by  .14  of  a  SD  in  reading  and  .09  of  a  SD 
mathematics. 

The  variables  related  to  language  (bilingual  and  LEP)  were  not  statistically  significant, 
implying  that  these  students  had  mastered  English  to  a  level  sufficient  enough  to  be 
legitimately  grouped  with  other  students  taking  the  TA  AS  that  year. 

Students  who  are  grouped  into  either  special  education  or  talented  and  gifted  categories 
had  significantly  different  achievement  level  scores  than  other  students.  Students  in  special 
education  had  significantly  lower  test  scores  than  other  students  by  nearly  half  of  a  SD.  The 
implication  here  is  that  the  average  abilities  of  students  in  special  education,  who  are  not 
exempt  from  the  TAAS,  are  not  at  the  same  level  as  other  students  taking  tire  TAAS.  In 
addition,  students  who  are  in  talented  and  gifted  programs  score  .51  of  a  SD  higher  than  other 
students  in  reading  and  .39  higher  in  mathematics. 

The  average  munber  of  weeks  students  spent  in  the  classroom  showed  small,  but 
significant  effects.  This  result  may  have  been  influenced  by  the  choice  of  sample,  since  students 
takmg  both  tests  are  likely  less  mobile.  The  extremely  small  coefficient  in  math  may  also  reflect 
the  fact  that  each  school  in  the  district  was  nnplementing  an  EvcKydciy  Mathetnatics  program, 
which  required  every  math  teacher  to  be  "on  the  same  page  at  the  same  time."  Thus,  students 
were  likely  to  face  less  discontinuity  in  mathematics  instruction  if  they  moved  to  otiier  schools 
within  the  district. 

The  strongest  predictor  of  4*  grade  scores  was  the  scores  students  achieved  in  3^^  grade. 

If  students  scored  higher  in  3^^  grade,  tiiey  were  also  likely  to  score  higher  in  4*  grade,  net  of  all 
other  student,  teacher,  classroom,  and  school  characteristics. 
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TEACHER-  AND  CLASSROOM-LEVEL  EFFECTS 

Overall,  none  of  the  variables  measuring  teacher  characteristics  in  these  model  have 
significant  effects  on  student  performance  at  the  second  level  of  analysis. 

Differences  between  classrooms  in  terms  of  their  demographic  make  up  seems  to  matter 
along  lines  similar  to  corresponding  student  level  effects.  One  interesting  exception  is  the  effect 
of  the  class  gender  composition  on  reading  scores.  At  the  individual  student  level,  we  saw  no 
difference  in  performance  between  boys  and  girls  in  fourth  grade.  At  the  classroom  level,  the 
results  suggest  that  classes  with  more  boys  than  girls  tend  to  have  lower  average  reading  scores 
by  about  3/10*=of  a  SD. 

In  addition,  classes  with  higher  proportions  of  students  in  the  white/ other  category  tend 
to  out-perform  classes  tiiat  are  predominantly  Latino.  The  proportion  of  students  in  the 
classroom  who  are  either  in  special  education  or  in  gifted  and  talented  education  is  also 
significant  in  both  models.  Classrooms  with  a  higher  proportion  of  students  who  are  older 
than  average  tend  to  have  lower  math  scores. 

SCHOOL-LEVEL  EFFECTS 

Average  TEA  rank  is  significant  (.35  of  an  SD  in  reading  and  .26  in  mathematics), 
suggesting  that  relative  school  quahty  is  an  important  factor  in  student  achievement.  Implicit 
is  a  confirmation  that  the  ranking  system  employed  by  TEA  has  indeed  identified  those  schools 
in  which  student  performance  on  average  has  not  improved  over  time. 

After  controlling  for  all  of  these  student,  classroom,  and  school  characteristics,  we  fail  to 
find  a  significant  effect  of  implementation  of  NAS  designs  in  San  Antonio.  This  same  result 
came  up  in  estimations  of  a  variety  of  other  model  specifications,  using  other  regression 
techniques  such  as  ordinary  least  squares,  three-level  linear  models  and  probit  models,  where 
the  dependent  variables  were  binary  indicators  of  passing  or  failing  scores. 

This  is  not  surprising  since  we  are  examining  effects  on  spring  1998  scores  and  many  of 
the  designs  had  not  been  in  place  that  long.  However,  note  that  the  focus  here  is  establishing  a 
baseline  in  the  district  to  set  the  stage  for  the  analyses  that  follow  of  classroom  conditions  in  the 
RAND  sample.  We  analyze  the  1997-1998  school  year  data  because  this  is  the  time  when 
instructional  effects  are  likely  to  occur  given  the  changes  that  we  described  in  the  districts 
between  1997-1998  and  1998-1999  school  years. 

GOODNESS  OF  HT 

To  establish  a  baseline  for  a  simple  measure  of  goodness  of  fit,  we  partitioned  the 
variance  in  student  test  scores  into  their  within-  and  between-classroom  components.  Prior  to 
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estimating  the  full  reading  and  mathematics  models,  a  two-level  ANOVA  regression  was  used 
to  estimate  within  and  between  classroom  variance  in  the  district  for  both  TAAS  reading  and 
mathematics  scores.  The  results  are  given  in  Table  6.3,  indicating  in  both  cases  that  about  17 
percent  of  the  total  variance  in  student  achievement  lies  between  classrooms  and  about  83 
percent  of  variance  within  classrooms. 

The  results  from  the  full  district-level  models  in  the  same  table  show  that  the  estimates 
explained  at  least  half  of  tire  variance  wiihin  classrooms.  While  the  reading  model  estimates 
explain  more  than  two-thirds  of  the  variation  in  test  scores  between  classrooms,  a  substantial 
portion  of  between-class  variation  remains  rmexplained.  Results  for  the  mathematics  model 
estimates  are  even  lower  as  only  about  45  percent  of  the  variance  between  classrooms  is 
explained. 

These  findings  concur  with  similar  analyses  of  student  scores  in  the  context  of  classrooms 
and  schools  (see  Lee,  Loeb,  and  Lubeck,  1998;  Gamoran,  1992;  Lee  and  Bryk,  1989).  The  results 
have  important  implications  for  educators  and  policymakers,  especially  within  a  context  of 
high  stakes  accountability  context  and  implementation  of  NAS  designs.  If  most  of  the 
differences  in  student  outcomes  come  from  within  classrooms  and  are  student-specific,  rather 
than  between  classrooms,  then  school  reform  efforts  which  focus  on  instructional  practices 
might  not  be  able  to  produce  desired  results. 


Table  6.3 

Variance  in  Student  Test  Scores  Explained  by  District  Sample  Models 


Reading 

Mathematics 

ANOVA 

Variance  within  Classrooms 

0.839 

0.839 

Variance  between  Classrooms 

0.169 

0.165 

District  Models 

Variance  within  Classrooms 

0.446 

0.422 

Variance  between  Classrooms 

0.056 

0.092 

Percent  of  variance  within 

classrooms  explained  by  the  model 

46.85 

49.74 

Percent  of  variance  between 
classrooms  explained  by  the  model 

66.96 

44.47 
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To  examine  the  impact  of  teaching  practices  and  classroom  environments  in  the  next 
section  we  attempt  to  further  control  for  variation  at  the  classroom  level  by  incorporating  data 
collected  from  the  teacher  survey  into  the  analysis. 

RAND'S  SURVEY  SAMPLE  DATA  AND  DEPENDENT  VARIABLES 

To  learn  more  about  the  effects  of  tire  classroom  environment  and  teaching  practices  on 
student  achievement  we  now  extend  the  models  previously  discussed  to  include  data  from  the 
teacher  survey  and  the  Stanford-9.  The  data  set  used  here  was  constructed  by  taking  a 
convenience  sample  of  fourth  grade  students  in  the  1998-1999  school  year  who,  in  addition  to 
valid  TAAS  reading  and  mathematics  scores,  had  valid  scores  on  the  RAND  administration  of 
the  Stanford-9,  and  were  in  classrooms  where  their  teachers  completed  the  1998  survey. 

In  addition  to  regressions  on  TAAS  reading  and  writing  scores,  a  third  model  takes  the 
Stanford-9  results,  reported  as  Normal  Curve  Equivalent  (NCE)  scores  as  the  dependent 
variable.31  Again,  for  ease  of  interpretation,  all  the  scores  are  standardized  to  have  a  mean  of 
zero  and  a  standard  deviation  of  one.^^ 

Classroom  level  variables,  such  as  the  percent  male  or  in  special  education,  from  the 
district  sample  were  used  in  this  analysis,  since  the  classroom  environments  would  not  change 
for  students  in  a  sub-sample. 

ADDITIONAL  INDEPENDENT  VARIABLES  FROM  THE  TEACHER  SURVEY 

Independent  variables  constructed  from  survey  responses  include  three  individual  items 
and  four  of  the  teacher  indices  discussed  in  Chapters  Four  and  Five.  One  additional  index  of 
related  survey  items  was  created  for  inclusion  in  the  quantitative  analysis. 

Key  to  our  analysis  is  the  inclusion  of  variables  related  to  instructional  conditions  in 
classrooms.  To  examine  the  effects  on  student  performance  of  the  extent  to  which  teachers 
employ  specific  strategies  we  construct  variables  reflecting  traditional  and  more  reform-like 
practices. 

The  first  three  variables  added  to  the  model  are  items  from  the  teacher  survey  viewed  as 
measures  of  traditional  instructional  strategies: 

•  Lecture, 

•  Students  work  individually  on  written  assignments /  worksheets  in  class,  and 

Stanford-9  Reading  NCE  scores  represent  a  student's  position  in  a  normal  distribution  of  Stanford-9 
raw  scores  nationwide.  The  NCE  scores  are  scaled  to  have  a  mean  of  50  and  a  standard  deviation  of  21.06 
and  do  not  correspond  precisely  to  percentile  rankings. 

No  further  transformations  were  required  for  the  Stanford-9  scores  as  die  distribution  fit  normality 
eissumptions  required  for  our  regression  techniques. 
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•  Students  practice  or  drill  on  computational  skills  that  we  would  expect  be  used  less 
frequently  within  the  context  of  educational  reform 

These  are  from  a  section  of  the  survey  that  asked  teachers  how  often  specific  instructional 
strategies  were  used  in  the  classroom.  Responses  were  ranked  on  a  5-point  scale  with  l=never, 
2=once  or  twice  a  semester,  3=once  a  twice  a  month,  4=once  or  twice  a  week,  and  5=almost 
every  day. 

The  reform  index,  described  in  Chapter  Five,  is  added  as  a  measure  of  reform-like 
instructional  strategies.  It  is  a  continuous  variable  consisting  of  responses  from  fifteen  items, 
standardized  to  have  a  mean  of  zero  and  a  standard  deviation  of  one  across  two  scales  (the  5- 
point  scale  mentioned  above  and  a  4-point  scale  that  ranges  from  l=almost  never  to  4=every 
lesson). 

Other  variables  are  constructed  from  indices  designed  to  measure  teachers'  perceptions 
of  iheir  students'  ability  and  readiness  to  learn,  and  general  factors  in  the  school  climate 
including: 

•  Factors  hindering  student  achievement:  Teachers  were  asked  to  rank  the  extent  to  which 
a  number  of  factors  might  hinder  their  students  from  achieving  at  high  levels  in 
schools.  These  include  lack  of  basic  skills,  inadequate  prior  student  preparation  in 
the  subject  area,  lack  of  student  motivation,  inadequate  support  from  parents  for 
students,  and  lack  of  student  discipline.  Each  was  ranked  on  a  4-point  scale  with 
l=greatly  and  4=not  at  all.  We  combined  teacher  responses  on  these  items  into  a 
summative  index  of  factors  hindering  student  achievement;  the  alpha  reliability  for 
this  index  was  .79,  and  tire  correlations  ranged  from  .26  to  .35. 

•  Collaboration:  A  continuous  variable  consisting  of  teacher  responses  to  six  items, 
standardized  to  have  a  mean  of  zero  and  a  standard  deviation  of  one  across  three 
scales  (6-points  ranging  from  l=does  not  describe  my  school  to  6=clearly  describes 
my  school;  4-points  ranging  from  l=greatly  improved  to  5=greatly  worsened, 
reverse  scaled  to  match  the  negative  to  positive  direction  of  the  other  scales;  and  4- 
points  ranging  from  l=strongly  disagree  to  4=strongly  agree)  (for  more  details  see 
Chapter  Four). 

•  Principal  leadership:  A  continuous  variable  consisting  of  teacher  responses  to  seven 
items  on  a  4-point  scale  ranging  from  l=strongly  disagree  to  4=strongly  agree  (for 
more  details  see  Chapter  Four). 

•  Quality  of  professional  development:  A  continuous  variable  consisting  of  teacher 
responses  to  ten  items  on  a  4-point  scale  ranging  from  l=strongly  disagree  to 
4=strongly  agree  (for  more  details  see  Chapter  Four). 
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MULTILEVEL  ANALYSIS  IN  THE  SURVEY  SAMPLE 

To  leam  more  about  the  effects  of  the  classroom  environment  and  teaching  practices  on 
achievement  scores,  we  extend  the  model  used  to  analyze  student  achievement  in  the  district 
by  including  the  eight  classroom  level  measures.  In  addition  to  modeling  the  TAAS  scores  as 
before,  we  also  analyze  Stanford-9  scores  for  the  sample  using  the  same  model  used  to 
investigate  the  effects  of  student  and  classroom  characteristics.  Thus,  we  are  able  to  examine 
whether  the  instructional  conditions  consistent  with  what  NAS  designs  are  intended  to 
promote  in  schools  are  related  to  student  achievement  on  a  high-stakes  tests  (e.g.,  TAAS 
reading)  as  well  as  an  independent  commercial  assessment  with  no  stakes  attached  (Stanford- 
9).  These  extensions  jdeld  the  following  changes  to  the  multilevel  models  described  above. 
Individual  Student  Model  (Level  1) 

At  the  student  level,  we  use  the  same  model  specification  as  in  the  district-wide  analysis 
for  each  of  the  TAAS  reading,  mathematics,  and  Stanford-9  regressions.  Within  the  Stanford-9 
model  the  1997  TAAS  readings  scores  are  used  to  control  for  prior  student  achievement. 

Yy  =  Poj  +  Pi  (Male)  +  P2( African  American)  +  p3(White  or  Other)  +  p4(Bilingual 

Education  Program)  +  PsCLimited  English  Proficiency)  +  PsfPree/ Reduced  Lunch 
Program  Eligible)  +  P7(Special  Education)  +  P8(Gifted  and  Talented  Education)  + 
pgfAge)  +  piofWeeks  with  Class)  +  Pii(1997  TAAS  Score)  +  P^CMissing  1997  TAAS 
Score)  +  ry, 
where 

•  Yy  is  the  dependent  variable,  i.e.,  die  TAAS  reading  or  mathematics  score  or  the 
Stanford-9  score  for  student  /  in  classroom  /; 

•  poj  is  the  level  1  constant  term  and  the  average  value  of  the  dependent  variable  in 
classroom  J; 

•  pi.i2  are  the  level  1  coefficients  for  the  listed  independent  variables; 

•  Tij  is  the  level  1  random  effect. 

Ctassroom  Context  Model  (Level  2) 

Poj=  Yo+ Yi(Male  Teacher)  +  Y2( African  American  Teacher)  +  YsCWhite  or  Other  Teacher) 

+  Y4(Teacher's  Years  of  Experience)  +  Y5(Master's  Degree  or  Higher)+  YefMale.;)  + 

Yvf  African  American.^)  +  YsCWhite  or  Other .y)  +  YgCBiUngual  Education  Program.^)  + 
YioCLimited  English  Proficiency .y)  +  Yii(Free/Reduced  Lunch  Program  EUgible.y)  + 
Yi2(Special  Education.y)  +  YufCifted  and  Talented  Education.y)  +  Yi4(Age.y)  + 
YisfWeeks  witii  Class.y)  +  TAAS  Score.y)  +  YnfMissing  1997  TAAS  Score.y) 

+  Yi8( Average  TEA  Rating,  1995  to  1997)  +  Yi9(Years  Implementing  a  NAS  Design) 
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+  y2o(Lecture)  +  y2i(Worksheets)  +  y22(Drill  on  Computational  Skills)  + 
Y23(Collaboration)  +  y24(Factors  Hindering  Student  Achievement)  +  y25(Principal 
Leadership)  +  y26(Quality  of  Professional  Development)  +  y27  (Reform)  +  u , 
where 

•  Poj  in  this  model  is  from  the  student  level  equation  above,  and  it  is  the  average 
test  score  in  classroom  j) 

•  To  is  the  intercept  term; 

•  Yi-27  are  the  level  2  coefficients  for  the  listed  independent  variables; 

•  Any  variable  X.j\s  the  classroom  average  of  student  characteristic  X-  and 

•  « is  a  level  2  random  effect. 

ANALYSIS  OF  THE  SURVEY  SAMPLE 

Table  6.4  gives  the  means  and  standard  deviations  for  all  variables  included  in  the 
Stanford-9  and  the  TAAS  reading,  and  mathematics  models  of  the  survey  sample.  Difference  of 
means  tests  revealed  no  statistically  significant  differences  in  student  or  teacher  characteristics 
between  die  district  and  the  survey  samples.  Since  classroom  averages  linked  to  students  in  the 
survey  sample  have  not  changed,  they  are  not  presented  here.  Compared  with  the  district  as  a 
whole,  the  survey  sample  had  slightly  higher  percentages  of  students  and  teachers  in  NAS 
schools  (approximately  60  percent  and  64  percent  compared  to  50  percent  and  49  percent, 
respectively),  given  the  nature  of  initial  decisions  as  to  which  schools  were  selected  to 
participate  in  the  teacher  survey  (compare  Table  6.4  with  6.1). 
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Table  6.4 

Descriptive  Statistics  of  1998  Sample  Data 


Variables 

Mean 

SD 

Dependent  Variables 

SAT  9  Reading  Score 

43.03 

21.55 

TAAS  Reading  Score,  1998 

76.54 

15.88 

TAAS  Mathematics  Score,  1998 

73.15 

15.21 

Independent  Variables  for  Students  (n  =  861) 

Percent  Male 

45.53 

Race  or  Ethnicity 

Percent  African  American 

10.92 

Percent  Asian/Pacific  Islander 

0.12 

Percent  Hispanic 

79.91 

Percent  Native  American 

0.23 

Percent  White 

8.71 

Percent  Free  /  Reduced  Lunch  program  Eligible 

88.85 

Percent  in  Special  Education 

10.10 

Percent  in  Gifted  and  Talented  Education 

7.78 

Percent  in  Bilingual  Education  program 

7.08 

Percent  with  limited  English  proficiency 

7.08 

1997  TAAS  Reading  Score  (n=730) 

69.45 

22.08 

1997  TAAS  Mathematics  Score  (n  =  713) 

72.54 

15.29 

Age  on  Test  Date  (4/30/1998) 

10.30 

0.44 

Number  of  Weeks  in  Classroom 

36.31 

6.63 

Percent  of  Students  in  NAS  Schools  (n  =  516) 

59.93 

Percent  in  Co-NECT  Schools  (n  =  71) 

8.25 

Percent  in  BLOB  Schools  (n  -  30) 

3.48 

Percent  in  MRSH  Schools  (n  -  133) 

15.45 

Percent  in  SFA/RW  Schools  (n  =  282) 

32,75 

Teacher  Characteristics  (n  =  63) 

Percent  Male 

7.94 

Total  Years  Teaching 

13.11 

8.71 

Teacher  race  or  ethnicity 

Percent  African  American 

12.70 

Percent  Asian/Pacific  Islander 

0.00 

Percent  Hispanic 

49.21 

Percent  Native  American 

0.00 

Percent  White 

38.10 

Percent  with  Master's  Degrees 

44.44 

Percent  Teaching  in  NAS  Schools  (n  =  40) 

63.49 

Percent  Teaching  in  Co-NECT  Schools  (n  =  6) 

9.52 

Percent  Teaching  in  ELOB  Schools  (n  =  2) 

3.17 

Percent  Teaching  in  MRSH  Schools  (n  =  10) 

15.87 

Percent  Teaching  in  SFA/RW  Schools  (n  =  22) 

34.92 
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_ Table  6.4  (cont.) _ 

Responses  to  Teacher  Survey  Questions  _ 

How  often  did  you  use  each  of  the  following  instructional  strategies  with 
your  class  last  year?  (1  =  Never,  5  =  Almost  every  day). 


Lecture  (n  =  62) 

4.44 

0.92 

Have  students  work  individually  on  written  assignments  /  worksheets  in 

class  (n  =  62) 

4.58 

0.56 

Have  students  practice  or  drill  computational  skills  (n  =  63) 

4.40 

0.66 

Teacher  Survey  Composites: 

Collaboration® 

0.00 

1.00 

Factors  Hindering  Student  Achievement  (1  =  Greatly  Worsened,  5  = 

Greatly  Improved) 

1.82 

0.63 

Principal  Leadership  Index  (1  =  Strongly  Disagree,  4  =  Strongly  Agree) 

2.32 

0.69 

Quality  of  Professional  Development  Index  (1  =  Strongly  Disagree,  4  = 

Strongly  Agree) 

2.82 

0.63 

Reform  Index® 

0.00 

1.00 

School  Characteristics  (n  -  23) 

1995-1997  Average  Texas  Education  Agency  Rating 

2.06 

0.26 

Years  Implementing  a  NAS  Design  (NAS  Schools  only  -  n  =  15) 

1.27 

0,46 

Percent  of  Schools  Implementing  a  NAS  Design  (n  =  15) 

65.22 

Percent  Implementing  Co-NECT  Design  (n  =  2) 

8.70 

Percent  Implementing  ELOB  Design  (n  =  1) 

4.35 

Percent  Implementing  MRSH  Design  (n  -  4) 

17.39 

Percent  Implementing  SFA/RW  Design  (n  =  8) 

34.78 

NOTES:  ^Constructed  with  mean  0  ,standard  deviation  1  to  homogenize  the  various  response 
formats  in  the  items  that  constitute  the  composite. 


MULTILEVEL  RESULTS  FOR  THE  SURVEY  SAMPLE 

To  focus  on  the  new  information  gained  from  extending  the  model,  we  report  only 
coefficient  estimates  for  the  teacher  survey  variables  (Table  6.5).  The  estimation  results  for  the 
rest  of  the  variables  in  the  models  revealed  few  changes  in  comparison  to  the  district  models. 
In  particular,  the  variables  relating  implementation  of  NAS  designs  to  student  outcomes  in  the 
survey  sample  were  not  statistically  significant. 

The  coefficient  estimates  summarized  in  Table  6.5  indicate  that  principal  leadership  does 
have  a  significant  effect  on  student  performance  but  only  on  the  state-mandated  TAAS  reading 
and  mathematics  tests.  One  implication  of  this  result  could  be  that  that  principals  are  reacting 
to  the  context  of  accountability  rather  than  increasing  student  achievement  and  focusing  their 
leadership  efforts  accordingly. 


Table  6.5 

Multilevel  Results  for  the  Relationships  between  Classroom  Conditions  and  199Srfide  Test  Scores 
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But  whether  looking  at  state-mandated  exams  or  the  independent  Stanford-9,  the 
extended  models  find  little  support  for  effects  of  either  reform-like  or  traditional 
instructional  strategies  alike,  net  of  other  student  and  classroom  level  factors.  The 
exception  is  the  significantly  positive  effect  on  TAAS  mathematics  scores  generated  by 
having  students  complete  assignments  or  worksheets  in  class.  This  finding  could 
possibly  be  a  consequence  of  the  district-mandated  Everyday  Mathematics  program, 
which  tends  to  emphasize  the  use  of  worksheets  in  workbooks. 

SUMMARY 

This  chapter  deepens  our  understanding  of  the  impact  of  NAS  designs  and 
classroom  practices  within  a  context  of  high  poverty  and  high-stakes  accountability 
through  quantitative  analysis  of  student  achievement  at  the  high  point  of  a  NAS 
implementation.  As  expected  because  of  the  early  stages  of  implementation,  NAS 
designs  had  no  significant  effects  on  student  achievement,  as  measured  by  TAAS 
reading  and  mathematics  scores.  However,  this  does  contrast  what  other  found  early 
on  in  the  implementation  process  in  other  settings  (see  Ross,  Wan,  Alberg,  Sanders, 
Write,  and  Stringfield,  forthcoming). 

To  further  inform  this  result,  we  examined  test  scores  for  a  sub-sample  of  students 
who  were  linked  to  teachers  in  our  survey  data.  Using  survey  data  to  control  further  for 
classroom  practices  and  teacher  perceptions  of  classroom  environments,  we  still  found 
no  statistically  significant  effect  from  implementation  of  NAS  designs. 

More  importantly,  we  did  not  find  that  instructional  conditions  promoted  by 
reforms  such  as  NAS— including  teacher-reported  collaboration,  quality  of  professional 
development,  and  reform-like  instructional  practices— were  related  to  student 
achievement  net  of  other  student  and  classroom  conditions. 

There  may  be  several  explanations  for  these  findings.  First,  the  instructional 
analyses  were  based  on  a  small  sample  size  of  63  classrooms,  and  this  small  sample  may 
have  not  been  powerful  enough  to  detect  classroom-level  effects  of  reform  like  practices 
(most  of  the  effects  of  reform  like  practices  in  Table  6.5  were  positive  in  sign  but 
statistically  insignificant).  Second,  the  levels  for  these  reform  like  practices  were  not  at 
extremely  high  levels  to  truly  test  whether  such  variation  was  related  to  student 
achievement.  We  attempted  to  provide  the  most  likely  case  for  such  effects  to  occur  by 
analyzing  1997-1998  data  when  instructional  conditions  tended  to  differ.  By  the 
following  year,  most  classrooms  were  more  similar  in  their  instruction,  which  was  more 
conventional  in  nature.  Third,  a  related  reason  is  that  because  of  the  pressure  of  the 


109 


district  to  improve  TAAS  achievement  scores  and  the  constraints  on  teacher  practice  of 
curricular  mandates,  teachers  may  have  reported  on  surveys  that  they  spent  time  on 
certain  reform-like  activities,  but  such  reports  in  reality  may  have  been  much  less 
reform-like.  Fourth,  it  may  be  that  the  teacher-reported  measures  here  were  not  well 
aligned  with  the  outcomes  examined,  which  is  critical  when  examining  instructional 
effects  on  achievement  (Cohen  and  Hill,  1998;  Gamoran  et  al.,  1997, 1995;  Newmann  et 
al.,  19%).  Studies  of  instruction  have  not  found  robust  effects  of  pedagogical  practices 
on  student  achievement  across  scores  (see  Mayer,  1999;  Gamoran  et  aL,  1995;  Burstein  et 
al.,  1995).  Certainly,  this  is  an  issue  that  deserves  further  research  particularly  in  schools 
and  classrooms  in  which  whole-school  designs  have  been  fully  implemented  and 
sustained  over  a  longer  period  of  time. 
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7.  IMPLICATIONS  FOR  SCHOOL  IMPROVEMENT  IN  HIGH-POVERTY 

SETTINGS 


Rather  than  reiterate  the  summary  of  findings  that  occur  within  each  chapter,  we 
focus  here  on  the  implications  of  school  improvement  efforts  such  as  NAS  in  high- 
poverty  settings. 

Currently,  tfirough  the  federal  funding  many  schools  ttiroughout  the  coimtry  are 
attempting  NAS-like  reforms  using  the  federal  funding  provided  by  such  programs  as 
Title  I  and  the  Comprehensive  School  Reform  Demonstration  (CSRD)  program.  Our 
study  in  conjimction  with  the  other  RAND  studies  on  NAS  has  clear  implications. 
Schools  attempting  comprehensive  school  reforms  face  many  obstacles  during 
implementation,  and  because  of  tihis,  whole-school  designs  face  continuing  challenges  in 
significantly  raising  the  achievement  of  all  students.  This  is  particularly  important  to 
remember  when  setting  expectations  for  school  improvement  under  new  federal,  state, 
and  local  programs— particularly  when  implementing  strategies  and  interventions  in 
high-poverty,  low-performing  settings. 

Because  the  target  of  the  federal  Title  I  and  CSRD  funds  is  primarily  high-poverty 
schools,  schools  most  likely  to  be  affected  by  the  CSRD  program  are  also  schools  that  are 
most  likely  to  face  very  fragmented  and  conflicting  enviroiunents,  difficult  and 
changing  political  currents,  new  accountability  systems,  trenchant  imions,  serious  lack 
of  slack  resources  in  terms  of  teacher  time,  and  demoralized  teachers  given  the 
fluctuating  reform  agenda,  and  the  difficult  task  of  improving  student  performance 
under  these  types  of  conditions  (for  a  description  of  CSRD  schools  see  Kirby  and 
Berends  et  al.,  in  review). 

Given  this,  federal  and  state  polic5miakers  need  to  think  critically  about  their 
current  stance  of  simultaneously  promoting:  high  stakes  testing;  the  implementation  of 
comprehensive  school  reforms  that  promote  innovative  curriculum  and  instructional 
strategies;  and  the  implementation  of  multiple  other  concurrent  reforms.  The 
implementation  of  high  stakes  testing  regimes— the  apparent  outcome  of  many 
standards  based  reforms— might  preclude  the  adoption  of  rich  and  varied  curricula  that 
challenge  students  and  motivate  them  toward  more  in-depth  learning  experiences.  It 
certainly  prevents  it  when  other  more  "skill  and  drill"  like  reforms  are  mandated  on  top 
of  the  design-based  curriculum.  The  current  study  shows  that  high  stakes  tests  are  a 
two  edged  sword  in  this  environment.  On  the  one  hand,  high  stakes  tests  motivate 
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schools  to  increase  performance  and  often  to  seek  out  new  curriculum  and  instructional 
strategies  associated  with  comprehensive  school  reforms.  On  the  other  hand,  tiiose  very 
same  tests  provide  disincentives  to  adopt  richer,  more  in-depth  curriculum. 

TOWARD  BETTER  EDUCATIONAL  POLICY 

Our  findings  are  consistent  with  Porter  and  Clune's  scheme  for  better  educational 
policy  (see  Porter  et  al.,  1988;  Porter,  1994;  Qune,  1998).  They  posit  that  educational 
policies  such  as  comprehensive  school  reform  are  likely  to  influence  teachers  and 
students  to  the  extent  to  which  they  are  specific,  powerful,  authoritative,  consistent,  and 
stable.  Specificity  ox  depth,  is  the  extent  to  which  the  comprehensive  school  reform 
provides  detailed  guidance  or  materials  to  help  schools  and  teachers  understand  what 
they  are  supposed  to  do  (e.g.,  materials  that  describe  the  stages  of  implementing  the 
design  and  ongoing,  clear  assistance  strategies  to  further  promote  implementation). 
Powerxeiexs  to  the  rewards  or  sanctions  attached  to  the  whole-school  reform,  such  as 
teachers  receiving  bonuses  or  greater  autonomy  if  they  comply  with  implementing  the 
design.  Authorityx^exs  to  the  degree  to  which  the  reform  policy  is  seen  as  legitimate 
and  as  having  the  support  oi  those  who  are  responsible  for  implementation.  If  respected 
groups  or  policymakers  have  strong  positive  views  toward  whole-school  reform  and  if 
teachers  support  its  implementation,  the  design  is  likely  to  have  greater  influence  in 
changing  teaching  and  learning.  Consistencyar  alignmentxoiexs  to  the  extent  to  which 
the  set  of  whole-school  interventions  and  strategies  are  aligned  with  a  common  mission 
and  vision,  both  within  the  school  and  the  district.  Stabilityxeiexs  to  the  reform  being 
sustained  over  time  in  a  coherent,  consistent  manner.  Pohcymakers  and  educators 
should  use  tiiese  dimensions  as  a  means  for  thinking  critically  about  tire  comprehensive 
school  reform  being  considered  and  whether  die  conditions  exist  for  it  to  flourish. 

Specificity 

In  the  case  of  San  Antonio,  the  specificity  of  the  designs  varied  as  did  the  detailed 
guidance  provided  by  the  district  and  design  teams.  The  district  was  supportive  of  the 
designs  and  promoted  their  adoption  throughout  the  district.  In  addition,  the  district 
devoted  a  great  deal  of  financial  resources  and  professional  development  to  schools  and 
teachers  to  change  the  learning  environments  for  San  Antonio  students.  Yet,  over  time, 
the  specificity  of  the  mathematics  and  reading  programs  far  outweighed  the  specificity 
of  the  designs.  As  shown  by  interviews  and  observations,  for  example,  elementary 
school  teachers  were  expected  to  be  within  a  certain  page  of  the  Everyday  Mathematics 
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workbooks,  and  district  staff  conducted  checks  in  schools  to  ensure  this  was  happening. 
Because  of  the  structure  of  these  other  curricular  programs  and  the  time  that  such 
prescriptiveness  took,  there  was  Utde  time  left  over  for  design  team  activities,  no  matter 
how  specific  those  may  have  been. 

Moreover,  about  a  third  of  the  NAS  teachers  in  our  longitudinal  teacher  sample 
reported  that  the  lack  of  coherent,  sustained  professional  development  hindered  their 
students'  success.  About  15  percent  of  non-NAS  teachers  reported  such  a  hindrance. 
This  contrast  suggests  that  the  specificity  of  the  district s  professional  development 
strategies  for  the  mix  of  reforms  was  overwhelming  to  NAS  teachers,  who  were 
implementing  the  entire  set  of  reforms  in  the  district. 

Even  the  design  teams  themselves  differ  in  the  level  of  specificity  that  they 
provide  schools.  On  the  one  hand,  SFA/RW  provides  specific  training  and  materials  to 
schools  to  structure  the  reading  program  for  elementary  level  students.  In  addition, 
assessments  are  provided  to  ensure  ffiat  grouping  arrangements  remain  flexible  and 
students  can  move  every  6-8  weeks  or  every  grading  period  to  a  new  group,  if 
necessary,  that  meets  their  educational  needs.  On  the  other  hand,  ELOB  offers  little 
specific  curricular  guidance,  providing  teachers  with  learning  principals  and  an 
emphasis  on  project-based  learning.  It  is  up  to  the  teachers  to  develop  the  curriculum 
for  the  learning  expeditions  and  all  the  xmits  that  comprise  them. 

As  we  foimd  across  all  the  designs,  teacher  training  declined  over  time,  and  the 
design  team  representatives  had  less  contact  with  schools  and  teachers  over  the  course 
of  this  study.  However,  teachers  desired  more  design  team  training  to  imderstand 
better  the  component  and  activities  of  the  designs.  They  reported  the  need  for  more 
concrete,  hands-on  training  that  would  enable  them  to  better  understand  design 
processes  and  activities. 

Power 

There  were  very  few  rewards  and  sanctions  associated  with  the  NAS  designs  in 
San  Antonio.  While  the  superintendent  had  desired  to  have  NAS  designs  be  the 
integrative  force  to  hold  the  district' s  reform  efforts  together,  ffiere  were  few  rewards  or 
sanctions  for  schools  and  teachers  to  adopt  and  implement  the  NAS  reform.  The 
stronger  rewards  and  sanctions  were  associated  with  the  TAAS,  not  implementation  of 
designs.  NAS  designs  were  merely  weak  tool  that  the  district  employed  to  bring  about 
school  change.  As  was  clearly  evident  by  the  1998-1999  school  year,  the  district  had 
invested  more  energy  in  its  mathematics,  reading,  and  language  arts  programs. 
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Certainly,  it  was  these  that  had  more  power  for  schools  and  teachers,  not  the  NAS 
designs. 

Thus,  if  districts  and  schools  want  to  have  whole-school  reforms  take  hold  in 
schools  and  be  sustained  over  time,  they  will  need  to  attend  to  the  rewards,  sanctions, 
and  incentives  associated  with  the  reform.  More  importandy,  they  will  need  to  attend 
to  the  rewards  and  sanctions  linked  to  other  reforms  and  assessments  to  vmderstand 
whether  implementation  of  whole-school  reforms  are  in  conflict  or  supported  by  the 
power  of  other  reforms. 

Authority 

During  the  time  of  this  study,  die  NAS  and  its  designs  were  all  seeking  greater 
authority.  At  the  time  of  this  study,  only  the  SFA  program  had  been  noted  as 
significandy  improving  students'  test  scores  (see  Herman  et  al.,  1999).  While  there  was 
some  achievement  evidence  coming  out  of  Memphis  (see  Ross  et  al.,  forthcoming),  this 
work  had  not  yet  been  published  in  a  peer-reviewed  journal  and  was  specific  to  one 
jurisdiction. 

As  our  interviews  with  teachers  and  district  staff  revealed,  the  district  had  hoped 
that  NAS  and  its  designs  would  be  the  glue  to  hold  the  district  together.  However, 
there  were  doubts  about  whether  the  designs  could  raise  students'  TAAS  scores  to  the 
extent  necessary  to  meet  the  state  accoimtability  requirements.  Hedging  its  bets,  the 
district  added  specific  curricular  programs  to  the  mix  of  reforms,  which  in  the  end, 
pushed  the  NAS  designs  to  the  periphery  of  the  everyday  lives  of  students  and  teachers 
in  classrooms. 

To  some  degree,  the  lack  of  authority  associated  with  the  NAS  reform  in  San 
Antonio  may  have  led  to  a  lack  of  teacher  support  for  the  designs.  We  found  that 
teacher  support  for  the  designs  declined  over  time  from  54  percent  in  1998  to  25  percent 
in  1999.  In  interviews,  teachers  expressed  that  they  doubted  whether  the  NAS  designs 
alone  helped  their  students  build  an  academic  fotmdation,  which  so  many  of  them  were 
lacking. 

Over  time,  as  some  designs  establish  more  of  an  evidentiary  base  for  their 
practices  and  approaches,  such  authority  will  appeal  to  schools  and  districts.  If  designs 
and  districts  can  integrate  professional  development  and  training  over  time,  it  is  likely 
that  ongoing  support  of  teachers  can  be  sustained  rather  than  wane. 
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Coherence  and  Alignment 

The  policy  of  the  NAS  reform  was  not  well  aligned  with  the  district-mandated 
curricular  programs.  The  district,  schools,  and  teachers  were  aware  of  this— a  pressing 
question  with  which  all  were  struggling.  In  fact,  during  the  time  of  our  study,  the 
district  was  pursuing  how  the  NAS  designs  could  be  integrated  with  the  mathematics, 
reading,  and  language  arts  programs.  This  was  certainly  a  difficult  issue  within  the 
high-stakes  environment. 

Yet,  while  the  district  and  teachers  were  living  this  question,  life  in  classrooms 
reflected  that  fourth  grade  students  spent  half  of  their  day  by  the  spring  of  1999  on  a 
very  specific  mathematics,  reading,  and  language  arts  curriculum.  Qearly,  district- 
issued  curriculum  and  instructional  strategies  limited  the  ability  of  teacher  to  develop 
units,  plan  expeditions  and  lessons,  and  engage  in  instructional  practices  described  in 
the  NAS  and  design  team  literature.  It  was  the  district  and  state  initiatives  that  directed 
the  educational  mission  of  all  schools  in  San  Antonio,  not  the  NAS  designs. 

Further  integration  and  forethought  on  behalf  of  districts  and  schools  to  integrate 
their  reform  strategies  and  interventions  is  a  challenge  in  many  environments, 
particularly  high-poverty,  low-performing  ones.  Yet,  to  align  the  reform  policies, 
strategies,  interventions,  and  supports  is  critical  for  meaningful  school  improvement 
that  continues  over  time. 

Stability 

After  the  data  collection  efforts  for  this  study  ended  in  the  spring  of  1999,  there 
was  a  critical  moment  that  occurred —the  superintendent' s  departure  and  decline  of  the 
NAS  initiative  in  San  Antonio.  As  so  often  happens  in  urban  school  districts,  the 
superintendent' s  position  was  tenuous  with  the  school  board.  After  a  school  board 
election  in  May,  1999,  two  new  board  members  were  elected.  Their  campaign  win 
tipped  the  balance  of  the  school  board.  Soon  after,  the  superintendent  accepted  a 
generous  buy-out  and  resigned.  Her  resignation  was  accepted  by  a  4-3  vote  by  the 
school  board.  Her  success  in  improving  the  test  scores  in  San  Antonio  schools  relative 
to  the  state,  yet  still  lagging  far  behind,  assured  her  a  position  elsewhere.  She  moved 
on. 

The  stability  of  the  NAS  and  its  designs  was  deeply  affected  by  the 
superintendent's  departure.  During  the  summer  of  1999,  teachers  were  asked  to  vote 

This  information  is  based  on  interviews  with  district  staff  and  some  design  teams  after 
the  superintendent  s  departure  as  well  as  reports  in  the  San  Antonio  Express-News. 
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whether  or  not  to  keep  the  designs.  Whereas  some  schools  initially  adopted  designs 
based  on  60  percent  of  their  teachers  voting  in  favor  of  adoption,  the  summer  1999  vote 
required  80  percent  of  the  teachers  to  vote  in  favor  of  adoption.  The  lack  of  support  that 
existed  in  NAS  schools,  the  overburden  that  NAS  teachers  reported,  and  the  uncertainty 
and  tension  within  the  district  likely  led  to  the  substantial  number  of  schools  that  voted 
to  drop  the  designs. 

The  stability  of  the  whole-school  improvement  efforts  with  NAS  designs  thus 
was  on  very  fragile  ground,  even  in  a  district  that  was  initially  so  supportive  of  NAS 
implementation. 

Unfortunately,  such  instability  is  all  too  common  in  high-poverty  settings.  To 
put  the  policy  pieces  in  place  for  reform  that  have  the  qualities  mentioned  here  and  to 
sustain  them  over  time,  despite  the  changes  in  leadership,  remains  a  significant 
challenge. 

SCHOOL  LEADERSHIP 

In  addition  to  the  importance  of  district  leadership  in  implementing  policies  that 
are  specific,  powerful,  authoritative,  aligned,  and  stable,  the  leadership  in  schools  is  also 
critical. 

Despite  all  that  was  going  in  the  district  and  the  overall  similarity  of  instructional 
conditions,  our  analysis  showed  that  principal  leadership  had  a  strong  effect  on  student 
achievement  scores.  Controlling  for  other  student,  teacher,  classroom,  and  school 
factors,  classrooms  in  which  teachers  reported  greater  principal  leadership  had  higher 
TAAS  reading  and  mathematics  scores  by  .15  and  .21  of  a  standard  deviation, 
respectively.  Principal  leadership  in  our  analysis  was  measured  by  teacher  reports 
about  principals  who  clearly  commxmicated  what  was  expected  of  teachers,  were 
supportive  and  encouraging  of  staff,  obtained  resources  for  the  school,  enforced  rules 
for  student  conduct,  talked  with  teachers  regarding  instructional  practices,  had 
confidence  in  the  expertise  of  the  teachers,  and  took  a  personal  interest  in  the 
professional  development  of  teachers. 

Our  previous  analyses  have  shown  the  importance  of  principal  leadership  in 
implementing  the  designs  (Berends  and  Kirby  et  al.,  2001;  Kirby  et  al.,  2001),  so  the  link 
between  implementation  and  performance  is  likely  to  be  important  as  more  and  more 
schools  adopt  comprehensive  school  reform.  Therefore,  the  importance  of  principal 
leadership  should  not  be  overlooked  when  adopting  and  implementing  whole-school 
reforms. 
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SUPPORT  FOR  SCHOOLWTOE  REFORM 

Our  analyses  show  that  the  design  teams  had  a  difficult  time  in  San  Antonio 
because  of  the  high-stakes  accoimtability  environment,  the  pressing  need  to  improve 
test  scores,  and  the  lack  of  the  qualities— depth/ specificity,  power,  authority, 
consistency/ alignment,  and  stabihty— of  the  policies  directly  related  to  design 
implementation. 

We  have  argued  that  the  implementation  of  designs  in  classrooms  was  affected 
by  factors  related  to  district,  schools,  teachers,  and  design  teams.  The  findings  here  are 
consistent  with  what  RAND  has  foimd  in  it  other  studies  of  NAS  (Kirby,  Berends,  and 
Naftel,  2001;  Bodilly,  1998;  2001;  Berends  and  Kirby  et  al.,  2001;  Berends,  2000;  Bodilly 
and  Berends,  1999).  Comprehensive  reforms  face  many  obstacles  during 
implementation,  and  because  of  this,  whole-school  designs  face  continuing  challenges  in 
significantly  raising  the  achievement  of  all  students,  particularly  tirose  in  high-poverty 
settings.  The  key  seems  to  be  implementing  the  reform  in  such  a  way  that  it  aligns  with 
the  other  school  reform  efforts  to  ensure  consistent  implementation  over  time.  The 
challenge  for  such  improvement  efforts  to  become  school  wide  looms  large  in  the 
current  environment  where  leadership  in  high-poverty  districts  and  schools  is  fragile 
and  unstable,  fraught  with  conflicting  policies  and  reform  efforts. 

As  policymakers  and  practitioners,  we  must  continue  to  grapple  with  the 
questions:  How  can  the  designs  become  schoolwide?  How  can  we  shape  policies 
related  to  comprehensive  school  reform  to  lead  to  meaningful  school  improvement 
efforts?  What  supports— of  states,  districts,  design  teams,  and  schools— are  necessary 
for  implementation  to  take  hold  and  be  sustained  over  time?  We  have  attempted  to 
shed  some  light  on  those  critical  aspects  of  policies  that  polic5nnakers  and  practitioners 
can  address  when  embarking  on  whole-school  reform  efforts. 

Time  will  tell  whether  districts,  schools,  design  teams,  teachers  and  NAS  can 
make  the  appropriate  adjustments  in  the  implementing  sites.  Additional  and  more 
specific  longitudinal  achievement  data  over  time  will  help  assess  the  conditions  under 
which  design-based  assistance  can  contribute  to  improving  student  performance. 
However,  as  this  study  has  emphasized,  it  is  critical  to  understand  the  local  conditions 
in  which  whole-school  reforms  operate.  Researchers  need  to  vmderstand  the  district 
reform  policies  and  how  principals  and  teachers  within  schools  enact  these  reforms. 
Moreover,  when  comparing  those  schools  that  me  implementing  reforms  with  schools 
that  are  not  (the  "control"  schools),  understanding  what  actually  is  being  implemented 
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in  the  comparison  schools  sheds  important  light  on  the  our  understanding  of  the  entire 
reform  effort.  In  high-poverty  settings,  then,  it  is  as  important  to  understand  reform 
efforts  in  the  "control"  schools  as  it  is  in  the  "treatment"  schools  to  provide  a  more 
comprehensive  assessment. 

Continued  research  is  critical,  then,  for  imderstanding  what  it  takes  to  sustain 
reform  efforts  such  as  NAS  and  other  comprehensive  school  reforms.  If  teachers, 
principals,  designs,  and  districts  can  sustain  their  focus  on  the  NAS  designs  to  structure 
the  educational  policies  and  their  enactment  in  schools  and  classrooms,  it  is  likely  that 
the  designs  will  become  not  only  more  widespread  across  schools,  but  also  deeper  in 
their  implementation  within  schools.  Perhaps,  they  may  even  be  sustained  over  a 
longer  period  of  time.  However,  accomplishing  that  goal  will  involve  better  policy, 
more  perseverance,  and  leadership  at  all  levels. 
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